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SERIES PREFACE 


n the Essentials of Behavioral Science seties, out goal is to provide readers with 

books that will deliver key practical information in an efficient, accessible style. 

The series features books on a variety of topics, such as statistics, psychologi- 
cal testing, and research design and methodology, to name just a few. For the ex- 
petienced professional, books in the series offer a concise yet thorough review of 
a specific area of expertise, including numerous tips for best practices. Students 
can turn to series books for a clear and concise overview of the important topics 
in which they must become proficient to practice skillfully, efficiently, and ethically 
in theit chosen fields. 

Wherever feasible, visual cues highlighting key points are utilized alongside sys- 
tematic, step-by-step guidelines. Chapters are focused and succinct. Topics are or- 
ganized for an easy understanding of the essential material related to a particular 
topic. Theory and research are continually woven into the fabric of each book, but 
always to enhance the practical application of the material, rather than to sidetrack 
or overwhelm readers. With this series, we aim to challenge and assist readers in 
the behavioral sciences to aspire to the highest level of competency by arming 
them with the tools they need for knowledgeable, informed practice. 

Essentials of Psychological Testing has three goals. The first is to survey the basic 
principles of psychometrics that test users need to use tests competently. The sec- 
ond goal is to supply, for each of these areas, the information required to under- 
stand and evaluate tests, and test use, at a basic level. The third goal is to provide 
readers who need or desire more comprehensive information on psychological 
testing with the major reference works in the field. It is important to emphasize 
that this volume is of intended to replace college courses in psychological testing, 
psychometrics, or tests and measurements. Rather, this book may serve as a review 
or refresher for those who have taken such courses, as well as a point of departure 
for further exploration of certain topics. For those who have not yet taken testing 
courses but are considering them, this volume may be useful as an orientation to 


ix 
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the field and as a preview. Finally, for those who are curious about psychological 
tests but do not intend to pursue formal studies in psychometrics, this book will 
acquaint them with the essential concepts of this field. 


Alan S. Kaufman, PhD, and Nadeen L. Kaufman, EdD, Founding Editors 
Yale University School of Medicine 


One 


INTRODUCTION TO PSYCHOLOGICAL 
TESTS AND THEIR USES 


he first and most general meaning of the term /est listed in the dictionary 

l is “a critical examination, observation, or evaluation.” Its closest synonym 

is rial. The word critical, in turn, is defined as “relating to ... a turning 

point or specially important juncture” Merriam-Webster’ Collegiate Dictionary, 

1995). No wonder, then, that when the term psychological appears in front of the 

word “est, the resulting phrase acquires a somewhat threatening connotation. Psy- 

chological tests are often used to evaluate individuals at some turning point or sig- 

nificant juncture in their lives. Yet, in the eyes of many people, tests seem to be 

trials on which too much depends and about which they know all too little. To a 

large extent, the purpose of this book is to give readers enough information about 

psychological tests and testing to remove their threatening connotations and to 

provide the means whereby consumers of psychological tests can gain more 
knowledge about their specific uses. 

Thousands of instruments can accurately be called psychological tests. Many more 
usurp the label either explicitly or by suggestion. The first objective of this book 
is to explain how to separate the former from the latter. Therefore, we start with 
the defining features that legitimate psychological tests of all types share. These 
features not only define psychological tests but also differentiate them from other 
kinds of instruments. 


PSYCHOLOGICAL TESTS 


A psychological testis a systematic procedure for obtaining samples of behavior, rel- 
evant to cognitive or affective functioning, and for scoring and evaluating those 
samples according to standards. A clarification of each of the main terms in this 
definition is vital to an understanding of all future discussion of tests. Rapid Ref- 
erence 1.1 explains the meaning and rationale of all the elements in the definition 
of a psychological test. Unless every condition mentioned in the definition is met, 
the procedure in question cannot accurately be called a psychological test. It is, 
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Basic Elements of the Definition of Psychological Tests 


Defining Element 


Explanation 


Rationale 


Psychological tests are 
systematic procedures. 


Psychological tests are 
samples of behavior. 


The behaviors sampled 
by tests are relevant to 
cognitive or affective func- 
tioning or both. 

Test results are evaluated 
and scored. 


To evaluate test results it 
is necessary to have stan- 
dards based on empirical 
data. 


They are characterized by 
planning, uniformity, and 
thoroughness. 


They are small subsets of 
much larger whole. 


m@ 


The samples are selected 
for their empirical or 
ractical psychological 
significance. 


19) 


Some numerical or cate- 
gory system is applied to 
test results, according to 
preestablished rules. 


There has to be a way of 
applying a common yard- 
stick or criterion to test 
results. 


Tests must be demonstra- 
bly objective and fair to 
be of use. 


Sampling behavior is effi- 
cient because the time 
available is usually limited. 


Tests, unlike mental 
games, exist to be of use; 
they are tools. 


There should be no ques- 
tion about what the re- 
sults of tests are. 


The standards used to 
evaluate test results lend 
the only meaning those 
results have. 


however, important to remember that in essence, psychological tests are simply 
behavior samples. Everything else is based on inferences. 

Psychological tests are often described as standardized for two reasons, both of 
which address the need for objectivity in the testing process. The first has to do 
with uniformity of procedure in all important aspects of the administration, scor- 
ing, and interpretation of tests. Naturally, the time and place when a test is ad- 
ministered, as well as the circumstances under which it is administered and the 
examiner who administers it, affect test results. However, the purpose of 
standardizing test procedures is to make all the variables that are under the con- 
trol of the examiner as uniform as possible, so that everyone who takes the test 
will be taking it in the same way. 

The second meaning of standardization concerns the use of standards for 
evaluating test results. These standards are most often norms derived from a 
group of individuals—known as the normative or standardization sample—in the 
process of developing the test. The collective performance of the standardization 
group or groups, both in terms of averages and variability, is tabulated and be- 
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comes the standard against which the 


performance of other individuals DON’T FORGET 


who take the test after it is standard- 
e The word test has multiple mean- 


ized will be gauged. ngs. 

Strictly speaking, the term ‘est * The term psychological test has a 
should be used only for those proce- very specific meaning, 
dures in which test takers’ responses e In this book, test will be used to re- 


fer to all instruments that fit the de- 


ate evaluated based on their correct- ere 
inition of psychological test. 


ness or quality. Such instruments al- 
qanay e Tests designed to sample skills, 

knowledge, or any other cognitive 

aspect of a person’s cognitive func- function will be referred to as ability 


tioning, knowledge, skills, or abilities. tests; all others will be labeled as 
personality tests. 


ways involve the appraisal of some 


On the other hand, instruments 
whose responses are neither evalu- 
ated nor scored as right-wrong or pass-fail are called inventories, questionnaires, sur- 
veys, checklists, schedules, ot projective techniques, and are usually grouped under the 
rubric of personality tests. These are tools designed to elicit information about a 
person’s motivations, preferences, attitudes, interests, opinions, emotional make- 
up, and characteristic reactions to people, situations, and other stimuli. Typically, 
they use questions of the multiple-choice or true-false type, except for projective 
techniques, which are open ended. They can also involve making forced choices 
between statements representing contrasting alternatives, or rating the degree to 
which one agrees or disagrees with various statements. Most of the time person- 
ality inventories, questionnaires, and other such instruments are of the self-report 
variety but some are also designed to elicit reports from individuals other than the 
person being evaluated (e.g., a parent, spouse, or teacher). For the sake of expe- 
diency, and following common usage, the term “est will be used throughout this 
book to refer to all instruments, regardless of type, that fit the definition of a psy- 
chological test. Tests that sample knowledge, skills, or cognitive functions will be 
designated as ability tests, whereas all others will be referred to as personality tests. 


Other Terms Used in Connection with Tests and Test Titles 


Some other terms that are used, sometimes loosely, in connection with tests bear 
explaining, One of these is the word scale, which can refer to 


e a whole test made up of several parts, for example, the Stanford-Binet 
Intelligence Scale; 
e a subtest, or set of items within a test, that measures a distinct and spe- 
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cific characteristic, for example, the Depression scale of the Minnesota 
Multiphasic Personality Inventory (MMPI); 

* an array of subtests that share some common characteristic, for ex- 
ample, the Verbal scales of the Wechsler intelligence tests; 

* a separate instrument made up of items designed to evaluate a single 
characteristic, for example, the Jaternal-External Locus of Control Scale 
(Rotter, 1966); or 

e the numerical system used to rate or to report value on some measured 
dimension, for example, a scale ranging from 1 to 5, with 1 meaning 
strongly disagree and 5 strongly agree. 


Thus, when used in reference to psychological tests, the term scale has become 
ambiguous and lacking in precision. However, in the field of psychological mea- 
surement—also known as psychometrics—scale has a mote precise meaning. It 
tefers to a group of items that pertain to a single variable and are arranged in or- 
der of difficulty or intensity. The process of arriving at the sequencing of the 
items is called scaling. 

Battery is another term often used in test titles. A battery is a group of several 
tests, or subtests, that are administered at one time to one person. When several 
tests are packaged together by a publisher to be used for a specific purpose, the 
word battery usually appears in the title and the entire group of tests is viewed as a 
single, whole instrument. Several examples of this usage occur in neuropsycho- 
logical instruments (such as the Halstead-Reitan Neuropsychological Battery) 
where many cognitive functions need to be evaluated, by means of separate tests, 
in order to detect possible brain impairment. The term battery is also used to des- 
ignate any group of individual tests specifically selected by a psychologist for use 
with a given client in an effort to answer a specific referral question, usually of a 
diagnostic nature. 


PSYCHOLOGICAL TESTS AS TOOLS 


The most basic fact about psychological tests is that they are tools. This means 
that they are always a means to an end and never an end in themselves. Like other 
tools, psychological tests can be exceedingly helpful—even irreplaceable—when 
used appropriately and skillfully. However, tests can also be misused in ways that 
may limit or thwart their usefulness and, at times, even result in harmful conse- 
quences. 

A good way to illustrate the similarities between tests and other, simpler, tools 
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is the analogy between a test and a 


hammer. Both are tools for specific DON veil FORGET 


urposes, but can be used in a varie 
PHE ; B4 Psychological tests are evaluated at 


of ways. A hammer is designed basi- two distinct points and in two different 
cally for pounding nails into various ways: 
surfaces. When used appropriately, |. When they are being considered 


as potential tools by prospective 
users; at this point, their technical 
qualities are of primary concern. 


skillfully, and for its intended purpose 
a hammer can help build a house, as- 


semble a piece of furniture, hang pic- 2. Once they are placed in use for a 

tures in a gallery, and do many other specific purpose; at this point, the 

things. Psychological tests are tools skill of the user and the way tests 

designed to help in drawing infer- T Ube crete Ra consist: 
ations. 


ences about individuals or groups. 
When tests are used appropriately and 
skillfully they can be key components in the practice and science of psychology. 

Just as hammers may be used for good purposes other than those for which 
they were intended (e.g., as paperweights or doorstops), psychological tests may 
also serve purposes other than those for which they were designed originally, 
such as increasing self-knowledge and self-understanding. Furthermore, just as 
hammers can hurt people and destroy things when used incompetently or mali- 
ciously, psychological tests can also be used in ways that do damage. When test 
results ate misinterpreted or misused, they can harm people by labeling them in 
unjustified ways, unfairly denying them opportunities, or simply discouraging 
them. 

All tools, be they hammers or tests, can be evaluated based on how well they 
are designed and built. When looked at from this point of view, prior to being 
used, tests are evaluated only in a limited, technical sense and their appraisal is of 
interest mostly to potential users. Once they are placed into use, however, tests 
cannot be evaluated apart from the skills of their users, the ways they are used, 
and the purposes for which they are used. This in-use evaluation often involves 
issues of policy, societal values, and even political priorities. It is in this context 
that the evaluation of the use of tests acquires practical significance for a wider 
range of audiences. 


Testing Standards 


Because of the unique importance of tests to all the professionals who use them 
and to the general public, since the mid-1950s, three major professional organi- 
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= zations have joined forces to promul- 

= hpi AEJEIENCE LZ gate standards that provide a basis for 
Ree eee ont ATE aoe E ann TA cvakanie tere: tecunerpmedces- and 
Testing Standards the effects of test use. The most re- 

cent version of these is the Standards 


e This designati ill b d fre- 
AE N EA for Educational and Psychological Testing , 


quently throughout this book to re- 


fer to the Standards for Educational published in 1999 by the American 
and P ee pee cues Educational Research Association 
ointly in by the American Ed- ai 

ucational Research Association, (AERA) and prepared jointly by 
American Psychological Associa- AERA, the American Psychological 
tion, and National Council on Mea- Association (APA), and the National 


surement in Education. Council on Measurement in Educa- 


* The Testing Standards are the single tion (NCME). As Rapid Reference 
most important source of criteria 


for the evaluation of tests, testing 1.2 indicates, these standards are 
practices, and the effects of test cited throughout this book and here- 
USE. 


after will be referred to as the Testing 
Standards. 


PSYCHOLOGICAL TESTS AS PRODUCTS 


The second most basic fact about psychological tests is that they are products. Al- 
though this is an obvious fact, most people are not mindful of it. Tests are prod- 
ucts primarily marketed to and used by professional psychologists and educators, 
just as the tools of dentistry are marketed and sold to dentists. The public at large 
remains unaware of the commercial nature of psychological tests because they are 
advertised through publications and catalogs targeted to the professionals who 
use them. Nevertheless, the fact remains that many, if not most, psychological 
tests are conceived, developed, marketed, and sold for applied purposes in edu- 
cation, business, or mental health settings. They also must make a profit for those 
who produce them, just like any other commercial product. 

As we will see, from the very beginning, the psychological testing enterprise 
was fueled principally by the need to make practical decisions about people. Since 
tests ate professional tools that can be used both to benefit people and as com- 
mercial products, some clarification of the various parties in the testing enterprise 
and their roles is justified. Rapid Reference 1.3 shows a list of the major partici- 
pants in the testing process and their roles. 

As the Testing Standards stipulate, “the interests of the various parties involved 
in the testing process are usually, but not always, congruent” (AERA, APA, 
NCME, 1999, p. 1). For example, test authors are usually, though not always, aca- 
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= fg Reference 1.3 


Participants in the Testing Process and Their Roles 


Participants Their Roles in the Testing Process 
Test authors and They conceive, prepare, and develop tests. They also 
developers find a way to disseminate their tests, by publishing them 


either commercially or through professional publications 
such as books or periodicals. 


Test publishers They publish, market, and sell tests, thus controlling their 
distribution. 
Test reviewers They prepare evaluative critiques of tests based on their 


technical and practical merits. 


Test users They select or decide to take a specific test off the shelf 
and use it for some purpose. They may also participate 
in other roles, e.g., as examiners or scorers. 


Test administrators or They administer the test either to one individual at a 


examiners time or to groups. 
Test takers They take the test by choice or necessity. 
Test scorers They tally the raw responses of the test taker and trans- 


form them into test scores through objective or me- 
chanical scoring or through the application of evaluative 
judgments. 


Test score interpreters They interpret test results to their ultimate consumers, 
who may be individual test takers or their relatives, 
other professionals, or organizations of various kinds. 


demicians or investigators who are mainly interested in psychological theorizing 
or research, rather than in practical applications or profits. Test users are most in- 
terested in the appropriateness and utility of the tests they use for their own pur- 
poses, whereas test publishers are naturally inclined to consider the profit to be 
made from selling tests foremost. Furthermore, participants in the testing pro- 
cess may perform one or more of all the various roles described in Rapid Refer- 
ence 1.3. Test users may administer, score, and interpret the results of tests they 
have selected or may delegate one or more of these functions to others under 
their supervision. Similarly, test publishers can, and often do, hire test developers 
to create instruments for which they think a market exists. Nevertheless, of all 
participants in the testing process, the Testing Standards assign “the ultimate re- 
sponsibility for appropriate test use and interpretation” predominantly to the test 
user (p. 112). 
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HISTORY OF PSYCHOLOGICAL TESTING 


Even though psychological tests can be used to explore and investigate a wide 
range of psychological variables, their most basic and typical use is as tools in mak- 
ing decisions about people. It is no coincidence that psychological tests as we know 
them today came into being in the early part of the 20th century. Prior to the rise of 
urban, industrial, democratic societies, there was little need for most people to 
make decisions about others, outside of those in their immediate families or close 
circle of acquaintances. In rural, agrarian, autocratic societies, major life decisions 
about individuals were largely made for them by parents, mentors, rulers and, above 
all, by the gender, class, place, and circumstances into which people were born. 
Nonetheless, well before the 20th century, there ate several interesting precursors 
of modern psychological testing within a variety of cultures and contexts. 


Antecedents of Modern Testing in the Occupational Realm 


A perennial problem in any field of employment is the question of how to select 
the best possible people for a given job. The oldest known precursors of psycho- 
logical testing are found precisely in this area, within the system of competitive ex- 
aminations developed in the ancient Chinese empire to select meritorious indi- 
viduals for government positions. This forerunner of modern personnel selection 
procedures dates back to approximately 200 B.C.E. and went through a number of 
transformations in its long history (Bowman, 1989). The Chinese civil service ex- 
aminations encompassed demonstrations of proficiency in music, archery, and 
horsemanship, among other things, as well as written exams in subjects such as 
law, agriculture, and geography. Apparently, the impetus for the development of 
this enlightened system of human resoutce utilization—open to any individual 
who was recommended to the emperor by local authorities throughout the em- 
pite—was the fact that China did not have the sort of hereditary ruling classes that 
were common in Europe until the 20th century. The Chinese imperial examina- 
tion system ended in 1905 and was replaced with selection based on university 
studies. In the meantime, however, that system served as an inspiration for the 
civil service exams developed in Britain in the 1850s, which, in turn, stimulated the 
creation of the US. Civil Service Examination in the 1860s (DuBois, 1970). 


Antecedents of Modern Testing in the Field of Education 


One of the most basic questions in any educational setting is how to ascertain that 
students have acquired the knowledge or expertise their teachers try to instill in 
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them. Thus, it is not surprising that the earliest use of testing within the realm of 
education occurred during the Middle Ages with the rise of the first universities 
in Europe in the 13th century. At about that time, university degrees came to be 
used as a means of certifying eligibility to teach, and formal oral examinations 
were devised to give candidates for degrees an opportunity to demonstrate their 
competence (DuBois, 1970). Little by little, the use of examinations spread to the 
secondary level of education and, as paper became cheaper and mote available, 
written examinations replaced the oral exams in most educational settings. By the 
late 19th century, in both Europe and the United States, examinations were a well- 
established method of ascertaining who should be awarded university degrees as 
well as who would be able to exercise a profession, such as medicine or law. 


Antecedents of Modern Testing in Clinical Psychology 


Another fundamental human question that can be and has been addressed by 
means of psychological testing is the problem of differentiating the “normal” 
from the “abnormal” within the intellectual, emotional, and behavioral atenas. 
However, in contrast to the occupational or educational contexts where the bases 
on which decisions are made have traditionally been fairly clear, the realm of psy- 
chopathology remained shrouded in mystery and mysticism for a much longer 
period. 

Several antecedents of psychological tests stem from the field of psychiatry 
(Bondy, 1974). Many of these early tests were developed in Germany in the sec- 
ond half of the 19th century, although some of them date from the early part of 
that century and stemmed from France. Almost invariably these instruments 
were devised for the express purpose of assessing the level of cognitive function- 
ing of patients with various kinds of disorders such as mental retardation or brain 
damage. Among the behavior samples used in these early tests were questions 
concerning the meaning of proverbs and the differences or similarities between 
pairs of words, as well as memory tasks such as the repetition of digit series pre- 
sented orally. Many of the techniques developed in the 19th century were inge- 
nious and survived to be incorporated into modern tests that are still in wide use 
(see McReynolds, 1986). 

In spite of their cleverness, developers of the early forerunners of clinical tests 
were handicapped by at least two factors. One was the dearth of knowledge—and 
the abundance of superstitions and misconceptions—concerning psychopathol- 
ogy. In this regard, for instance, the distinction between psychosis and mental re- 
tardation was not even clearly formulated until 1838, when the French psychia- 
trist Esquirol suggested that the ability to use language is the most dependable 
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criterion for establishing a person’s level of mental functioning. A second factor 
preventing the widespread dissemination and use of the early psychiatric tests 
was their lack of standardization in terms of procedures or of a uniform frame of 
reference against which to interpret results. To a large extent, the techniques de- 
veloped by 19th-century neurologists and psychiatrists like Guislain, Snell, von 
Grashey, Rieger, and others were devised for the purpose of examining a specific 
patient or patient population. These behavior samples were collected in an un- 
systematic fashion and were interpreted by clinicians on the basis of their pro- 
fessional judgment rather than with reference to normative data (Bondy, 1974). 

A significant breakthrough was achieved in psychiatry during the 1890s, when 
Emil Kraepelin set out to classify mental disorders according to their causes, 
symptoms, and courses. Kraepelin wanted to bring the scientific method to bear 
on psychiatry and was instrumental in delineating the clinical picture of schizo- 
phrenia and bipolar disorder, which—at the time—were known respectively as 
dementia praecox and manic-depressive psychosis. He proposed a system for comparing 
sane and insane individuals on the basis of characteristics such as distractibility, 
sensitivity, and memory capacity and even pioneered the use of the free- 
association technique with psychiatric patients. Although some of Kraepelin’s stu- 
dents devised a battery of tests and continued to pursue the goals he had set out, 
the results of their work were not as fruitful as they had hoped (DuBois, 1970). 


Antecedents of Modern Testing in Scientific Psychology 


The investigations of the German psychophysicists Weber and Fechner in the 
mid-19th century initiated a series of developments that culminated in Wilhelm 
Wundt’s creation of the first laboratory dedicated to research of a purely psycho- 
logical nature in Leipzig, Germany, in 1879. This event is considered by many as 
the beginning of psychology as a separate, formal discipline, apart from philoso- 
phy. With the rise of the new discipline of experimental psychology, there also 
arose much interest in developing apparatus and standardized procedures for 
mapping out the range of human capabilities in the realm of sensation and per- 
ception. The first experimental psychologists were interested in discovering gen- 
eral laws governing the relationship between the physical and psychological 
worlds. They had little or no interest in individual differences—the main item of 
interest in differential psychology and psychological testing—which they, in fact, 
tended to view as a source of error. Nevertheless, their emphases on the need for 
accuracy in their measurements and for standardized conditions in the lab would 
prove to be important contributions to the forthcoming field of psychological 
testing. 
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Wundt’s German lab flourished in the last decades of the 19th century and 
trained many psychologists from the United States and elsewhere who would go 
back to theit countries to establish their own similar labs. At about the same time, 
an Englishman named Francis Galton became interested in the measurement of 
psychological functions from an entirely different perspective. Galton was a man 
of great intellectual curiosity and many accomplishments, whose privileged social 
and financial position alowed him to pursue a wide range of interests. He was also 
a cousin and a great admirer of Charles Darwin, whose theory of evolution of 
species by natural selection had revolutionized the life sciences in the mid-19th 
century. After reading his cousin’s treatise on the origin of species, Galton de- 
cided to pursue his interest in the notion that intellectual gifts tend to run in fam- 
ilies. To this end he set up an anthropometric lab in London, where for several 
years he collected data on a number of physical and physiological characteris- 
tics—such as arm span, height, weight, vital capacity, strength of grip, and sen- 
sory acuity of various kinds—on thousands of individuals and families. Galton 
was convinced that intellectual ability was a function of the keenness of one’s 
senses in perceiving and discriminating stimuli, which he in turn believed was 
hereditary in nature. Through the accumulation and cross-tabulation of his an- 
thropometric data, Galton hoped to establish both the range of variation in these 
characteristics, as well as their interrelationships and concordance across individ- 
uals with different degrees of familial ties (Fancher, 1996). 

Galton did not succeed in his ultimate objective, which was to promote eugen- 
ics, a field of endeavor he had originated that aimed at improving the human race 
through selective breeding of its ablest specimens. To this end, he wanted to de- 
vise a way of assessing the intellectual capacity of children and adolescents 
through tests so as to identify the most gifted individuals early and encourage 
them to produce many offspring. Nevertheless, Galton’s work was continued and 
considerably extended in the United States by James McKeen Cattell, who also 
tried, fruitlessly, to link various measures of simple discriminative, perceptive, 
and associative power (which he labeled “mental” tests) to independent estimates 
of intellectual level, such as school grades. 

In light of some events of the 20th century, such as those in Nazi Germany, 
Galton’s aim seems morally offensive to most contemporary sensibilities. How- 
ever, at the time he coined the term eugenics and enunciated its aims, the genocidal 
potential of this endeavor was not generally perceived, and many illustrious indi- 
viduals of that era were enthusiastic eugenicists. In the process of his pursuit, 
however misguided it may seem to us today, Galton did make significant contri- 
butions to the fields of statistics and psychological measurements. While chart- 
ing data comparing parents and their offspring, for instance, he discovered the 
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phenomena of regression and correlation, which provided the groundwork for 
much subsequent psychological research and data analyses. He also invented de- 
vices for the measurement of hearing acuity and weight discrimination, and initi- 
ated the use of questionnaires and word association in psychological research. As 
if these accomplishments were not enough, Galton also pioneered the twin-study 
method that, once refined, would become a primary research tool in behavior ge- 
netics. 

One additional contribution to the nascent field of psychological testingin the 
late 1800s deserves mention because it would lead directly to the first successful 
instrument of the modern era of testing. While studying the effects of fatigue on 
children’s mental ability, the German psychologist Hermann Ebbinghaus—best 
known for his groundbreaking research in the field of memory—devised a tech- 
nique known as the Ebbinghaus Completion Test. This technique called for chil- 
dren to fill in the blanks in text passages from which words or word-fragments 
had been omitted. The significance of this method, which would later be adapted 
for a variety of different purposes, is twofold. First, because it was given to whole 
classes of children simultaneously, it foreshadowed the development of group 
tests. What is more important, however, is that the technique proved to be an ef- 
fective gauge of intellectual ability, as the scores derived from it corresponded 
well with the students’ mental ability as determined by rank in class. As a result of 
this, Alfred Binet was inspired to use the completion technique and other com- 
plex mental tasks in developing the scale that would become the first successful 
intelligence test (DuBois, 1970). 


The Rise of Modern Psychological Testing 


By the early 1900s everything necessary for the rise of the first truly modern and 
successful psychological tests was in place: 


° Laboratory tests and tools generated by the early experimental psychol- 
ogists in Germany, 

e Measurement instruments and statistical techniques developed by Gal- 
ton and his students for the collection and analysis of data on individual 
differences, and 

e An accretion of significant findings in the budding sciences of psychol- 
ogy, psychiatry, and neurology. 


All of these developments provided the foundation for the rise of modern test- 
ing. The actual impetus for it, however, came from the practical need to make de- 
cisions in educational placement. 
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In 1904, the French psychologist Alfred Binet was appointed to a commission 
charged with devising a method for evaluating children who, due to mental retar- 
dation or other developmental delays, could not profit from regular classes in the 
public school system and would require special education. Binet was particularly 
well prepared for this task, as he had been engaged in investigating individual dif- 
ferences by means of a variety of physical and physiological measures, as well as 
tests of more complex mental processes, such as memory and verbal compre- 
hension. In 1905, Binet and his collaborator, Theodore Simon, published the first 
useful instrument for the measurement of general cognitive abilities or global in- 
telligence. The 1905 Binet-Simon scale, as it came to be known, was a seties of 30 
tests or tasks varied in content and difficulty, designed mostly to assess judgment 
and reasoning ability irrespective of school learning. It included questions deal- 
ing with vocabulary, comprehension, differences between pairs of concepts, and 
so on, as well as tasks that included repeating series of numbers, following direc- 
tions, completing fragmentary text passages, and drawing. 

The Binet-Simon scale was successful because it combined features of earlier 
instruments in a novel and systematic fashion. It was more comprehensive in its 
coverage than earlier instruments devoted to evaluating narrower abilities. It was, 
in fact, a small battery of carefully selected tests arranged in order of difficulty and 
accompanied by precise instructions on how to administer and interpret it. Binet 
and Simon administered the scale to 50 normal children ranging in age from 3 to 
11 years, as well as to children with various degrees of mental retardation. The re- 
sults of these studies proved that they had devised a procedure for sampling cog- 
nitive functioning whereby a child’s general level of intellectual ability could be 
described quantitatively, in terms of the age level to which her or his performance 
on the scale corresponded. The need for such a tool was so acute that the 1905 
scale would be quickly translated into other languages and adapted for use out- 
side France. 


The Birth of the IQ 

Binet himself revised, expanded, and refined his first scale in 1908 and 1911. Its 
scoring developed into a system in which credit for items passed was given in 
terms of years and months so that a mental evel could be calculated to represent 
quality of performance. In 1911 a German psychologist named William Stern 
proposed that the mental level attained on the Binet-Simon scale, relabeled as a 
mental age score, be divided by the chronological age of the subject to obtain a men- 
tal quotient that would more accurately represent ability at different ages. To 
eliminate the decimal, the mental quotient was multiplied by 100, and soon be- 
came known as the żnzelligence quotient, or IQ. This now-familiar score, a rue ratio 
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IQ, was popularized through its use in the most famous revision of the Binet- 
Simon scales—the Stanford-Binet Intelligence Scale—published in 1916 by 
Lewis Terman. In spite of several problems with the ratio IQ, its use would last 
for several decades, until a better way of integrating age into the scoring of intel- 
ligence tests (described in Chapter 3) was devised by David Wechsler (Kaufman, 
2000; Wechsler, 1939). Binet’s basic idea—namely, that to be average, below av- 
erage, or above average in intelligence means that one performs at, below, or 
above the level typical for one’s age group on intelligence tests—has survived and 
become one of the primary ways in which intelligence is assessed. 

While Binet was developing his scales in France, in England, Charles Spear- 
man (a former student of Wundt’s and follower of Galton) had been trying to 
prove empirically Galton’s hypothesis concerning the link between intelligence 
and sensory acuity. In the process he had developed and expanded the use of cor- 
relational methods pioneered by Galton and Karl Pearson, and provided the con- 
ceptual foundation for factor analysis, a technique for reducing a large number of 
variables to a smaller set of factors that would become central to the advancement 
of testing and trait theory. 

Spearman also devised a theory of intelligence that emphasized a general 
intelligence factor (or g) present in all intellectual activities (Spearman, 1904a, 
1904b). He had been able to gather moderate support for Galton’s notions by 
correlating teachers’ ratings and grades with measures of sensory acuity, but soon 
realized that the tasks assembled in the Binet-Simon scale provided a far more 
useful and reliable way of assessing intelligence than the tools he had been using, 
Even though Spearman and Binet differed widely in theit views about the nature 
of intelligence, their combined contributions are unsurpassed in propelling the 
development of psychological testing in the 20th century. 


Group Testing 

At the time Binet died, in 1911, he had already considered the possibility of adapt- 
ing his scale to other uses and developing group tests that could be administered 
by one examiner to large groups for use in the military and other settings. The ful- 
fillment of that idea, however, would not take place in France but in the United 
States, where the Binet-Simon scale had been rapidly translated and revised for 
use primarily with schoolchildren and for the same purpose as it had been devel- 
oped in France. 

Upon the entry of the United States into World War I in 1917, the APA pres- 
ident, Robert Yerkes, organized a committee of psychologists to help in the war 
effort. It was decided that the most practical contribution would be to develop 
a group test of intelligence that could be efficiently administered to all recruits 
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into the U.S. Army, to help in making personnel assignments. The committee, 
made up of leading test experts of the day, including Lewis Terman, hastily as- 
sembled and tried out a test that came to be known as the Army Alpha. It con- 
sisted of eight subtests measuring verbal, numerical, and reasoning abilities, as 
well as practical judgment and general information. The test, which would even- 
tually be administered to more than a million recruits, made use of materials 
from various other instruments, including the Binet scales. In constructing it, 
the committee relied heavily on an unpublished prototype group test developed 
by Arthur Otis, who had devised multiple-choice items that could be scored ob- 
jectively and rapidly. 

The Army Alpha proved to be extremely useful. It was followed rapidly by the 
Army Beta, a supposedly equivalent test that did not require reading and could 
thus be used with recruits who were illiterate or non—English speaking, Unfortu- 
nately, the haste with which these tests were developed and put into use resulted 
in a number of inappropriate testing practices. In addition, unwarranted conclu- 
sions were made on the basis of the massive amounts of data that quickly accu- 
mulated (Fancher, 1985). Some of the negative consequences of the ways in 
which the Army testing program, and other massive testing efforts from that era, 
were implemented damaged the reputation of psychological testing in ways that 
have been difficult to surmount. Nevertheless, through the mistakes that were 
made early in the history of modern testing, a great deal was learned that later 
served to correct and improve the practices in this field. Furthermore, with the 
Army tests the field of psychology decisively stepped out of the lab and academic 
settings and demonstrated its enormous potential to contribute to real-world 
applications. 

After World War I, psychological testing came into its own in the United 
States. Otis published his Group Intelligence Scale, the test that had served as a 
model for the Army Alpha, in 1918. E. L. Thorndike, another important Ameti- 
can pioneer working at Teachers College at Columbia, produced an intelligence 
test for high school graduates, standardized on a more select sample (namely, 
college freshmen) in 1919. From then on, the number of published tests grew 
rapidly. Procedural refinements were also swiftly instituted in test administration 
and scoring. For example, test items of different types began to be presented in a 
mixed order rather than as separate subtests so that an overall time limit could be 
used for a test, eliminating the need for separate timing of subtests. Issues of stan- 
dardization, such as eliminating words that could be read with different pronun- 
ciations in spelling tests, came to the fore, as did tests’ ¢rustworthiness—a term that, 
at that time, encompassed what is currently meant by reliability and validity (Du- 
Bois, 1970). 
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OTHER DEVELOPMENTS IN PSYCHOLOGICAL TESTING 


The successes achieved with the Binet and Army tests proved their worth in help- 
ing to make decisions about people. This soon led to efforts to devise instru- 
ments to help in different kinds of decisions. Naturally, the settings where an- 
tecedents of psychological tests had arisen—schools, clinics, and psychology 
labs—also gave rise to the new forms and types of modern psychological tests. 

A thorough review of the history of testing in the first half of the 20th century 
is beyond the scope of this work. Nevertheless, a brief summary of the most 
salient developments is instructive both for its own sake and to illustrate the di- 
versity of the field, even in its early phase. 


Standardized Testing in Educational Settings 


As the number of people availing themselves of educational opportunities at all 
levels grew, so did the need for fair, equitable, and uniform measures with which 
to evaluate students at the beginning, middle, and final stages of the educational 
process. Two major developments in standardized educational testing in the early 
part of the 20th century are highlighted in the ensuing paragraphs. 


Standardized Achievement Tests 

Pioneered by E. L. Thorndike, these measures had been under development 
since the 1880s, when Joseph Rice began his attempts to study the efficiency of 
learning in schools. Thorndike’s handwriting scale, published in 1910, broke new 
ground in creating a series of handwriting specimens, ranging from very poor to 
excellent, against which subjects’ performance could be compared. Soon after, 
standardized tests designed to evaluate arithmetic, reading, and spelling skills 
would follow, until measures of these and other subjects became a staple of ele- 
mentary and secondary education. Today, standardized achievement tests are 
used not only in educational settings, but also in the licensing and certification of 
professionals who have completed their training. They are also used in other sit- 
uations, including personnel selection, that require the assessment of mastery of 
a given field of knowledge. 


Scholastic Aptitude Tests 

In the 1920s objective examinations, based loosely on the Army Alpha test, be- 
gan to be used in addition to high school grades for the purpose of making ad- 
missions decisions in colleges and universities. This momentous development, 
which culminated in the creation of the Scholastic Aptitude Test (SAT) in 1926, 
foreshadowed the arrival of many more instruments that are used to select can- 
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The Big Test 


Nicholas Lemann’s (1999) book The Big Test: The Secret History of the American 
Meritocracy uses college admissions testing programs, specifically the SAT, to illus- 
trate the intended and unintended consequences that such testing programs can 
have for society. The large-scale use of standardized test scores for deciding on ad- 
missions into leading institutions of higher education was pioneered by James 
Bryant Conant, president of Harvard University, and Henry Chauncey, the first 
president of the Educational Testing Service (ETS), in the 1940s and 1950s. Their 
goal was to change the process whereby access to these institutions—and to the 
positions of power that usually accrue to those who attend them—is gained from 
one based on wealth and social class to one based mainly on ability as demon- 
strated through test scores. Lemann maintains that although this use of testing did 
open up the doors of higher education to children of the middle and lower socio- 
economic classes, it also generated a new meritocratic elite that perpetuates itself 
across generations and largely excludes the children of underprivileged racial mi- 
norities who lack the early educational opportunities needed to succeed on the 
tests. 


didates for graduate and professional schools. Among the best known examples 
of tests of this type are the Graduate Record Exam (GRE), Medical College Ad- 
mission Test (MCAT), and Law School Admission Test (LSAT), used by doctoral 
programs, medical schools, and law schools, respectively. Although each of these 
tests contains portions specific to the subject matter of its field, they also typically 
share a common core that emphasizes the verbal, quantitative, and reasoning 
abilities needed for success in most academic endeavors. Interestingly, although 
their purpose is different from that of the standardized achievement tests, their 
content is often similar. Rapid Reference 1.4 presents information about a fasci- 
nating account of the history of higher education admissions testing in the United 
States. 


Personnel Testing and Vocational Guidance 


The optimal utilization of people’s talents is a major goal of society to which psy- 
chological testing has been able to contribute in important ways almost from its 
beginnings. Decisions concerning vocational choice need to be made by individ- 
uals at different points in their lives, usually during adolescence and young adult- 
hood but also increasingly at midlife. Decisions concerning the selection and 
placement of personnel within business, industry, and military organizations 
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need to be made on an ongoing basis. Some of the main instruments that came 
into being early and have proved to be particularly helpful in making both of these 
kinds of decisions are described in the following sections. 


Tests of Special Skills and Aptitudes 

The success of the Army Alpha test stimulated interest in developing tests to se- 
lect workers for different occupations. At the same time, applied psychologists 
had been working out and using a basic set of procedures that would justify the 
use of tests in occupational selection. Basically, the procedures involved (a) iden- 
tifying the skills needed for a given occupational role by means of a ob analysis, (b) 
administering tests designed to assess those skills, and (c) correlating the test re- 
sults with measures of job performance. Using variations of this procedure, from 
the 1920s on, psychologists were able to develop instruments for selecting 
trainees in fields as diverse as mechanical work and music. Tests of clerical, spa- 
tial, and motor abilities soon followed. The field of personnel selection in indus- 
try and the military grew up around these instruments, along with the use of job 
samples, biographical data, and general intelligence tests of the individual and 
group types. Many of the same instruments have also been used profitably in 
identifying the talents of young people seeking vocational guidance. 


Multiple Aptitude Batteries 

The use of tests of separate abilities in vocational counseling would largely give 
way in the 1940s to multiple aptitude batteries, developed through the factor an- 
alytic techniques pioneered by Spearman and expanded in England and the 
United States through the 1920s and 1930s. These batteries are groups of tests, 
linked by a common format and scoring basis, that typically profile the strengths 
and weaknesses of an individual by providing separate scores on various factors 
such as verbal, numerical, spatial, logical reasoning, and mechanical abilities, 
rather than the single global score provided by the Binet and Army test IQs. Mul- 
tiple aptitude batteries came into being following the widespread realization, 
through factor analyses of ability test data, that intelligence is not a unitary con- 
cept and that human abilities comprise a broad range of separate and relatively in- 
dependent components or factors. 


Measures of Interests 

Just as tests of special skills and aptitudes arose in industry and later found some 
use in vocational counseling, measures of interests originated for the purpose of 
vocational guidance and later found some use in personnel selection. Truman L. 
Kelley, in 1914, produced a simple Interest Test, possibly the first interest inven- 
tory ever, with items concerning preferences for reading materials and leisure ac- 
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tivities as well as some involving knowledge of words and general information. 
However, the breakthrough in this particular area of testing took place in 1924, 
when M. J. Ream developed an empirical key that differentiated the responses of 
successful and unsuccessful salesmen on the Carnegie Interest Inventory devel- 
oped by Yoakum and his students at the Carnegie Institute of Technology in 1921 
(DuBois, 1970). This event marked the beginning of a technique known as empir- 
ical criterion keying, which, after refinements such as cross-validation procedures 
and extensions to other occupations, would be used in the Strong Vocational In- 
terest Blank (SVIB), first published in 1927, and in other types of inventories as 
well. The current version of the SVIB—called the Strong Interest Inventory® 
(SII)—is one of the most widely used interest inventories and has been joined by 
many more instruments of this type. 


Clinical Testing 


By the start of the 20th century the field of psychiatry had embarked on more sys- 
tematic ways of classifying and studying psychopathology. These advances pro- 
vided the impetus for the development of instruments that would help diagnose 
psychiatric problems. The main examples of this type of tools are discussed here. 


Personality Inventories 

The first device of this kind was the Woodworth Personal Data Sheet (P-D 
Sheet), a questionnaire developed during World War I to screen recruits who 
might suffer from mental illnesses. It consisted of 116 statements regarding feel- 
ings, attitudes, and behaviors obviously indicative of psychopathology to which 
the respondent answered simply yes or no. Although the P-D Sheet showed some 
promise, World War I ended before it was placed into operational use. After the 
wat there was a period of experimentation with other, less obvious, kinds of items 
and with scales designed to assess neuroticism, personality traits—such as intro- 
version and extraversion—and values. Innovations in the presentation of items 
aimed at reducing the influence of social desirability, like the forced-choice tech- 
nique introduced in the Allport-Vernon Study of Values in 1931, came into being. 
However, the most successful personality inventory of that era, and one which 
still survives today, was the Minnesota Multiphasic Personality Inventory 
(MMPI; Hathaway & McKinley, 1940). The MMPI combined items from the 
P-D Sheet and other inventories, but used the empirical criterion keying tech- 
nique pioneered with the SVIB. This technique resulted in a less transparent in- 
strument on which respondents could not dissemble as easily because many of 
the items had no obvious reference to psychopathological tendencies. 
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Since the 1940s, personality inventories have flourished. Many refinements 
have been introduced in their construction, including the use of theoretical per- 
spectives—such as Henry Murray’s (1938) system of needs—and internal consis- 
tency methods of selecting items. Furthermore, factor analysis, which had been so 
crucial to the study and differentiation of abilities, also began to be used in per- 
sonality inventory development. In the 1930s, J. P. Guilford pioneered the use of 
factor analysis to group items into homogeneous scales while, in the 1940s, R. B. 
Cattell applied the technique to try to identify the personality traits that are most 
pivotal and, therefore, worthy of investigation and assessment. Currently, factor 
analysis plays an integral role in most facets of test theory and test construction. 


Projective Techniques 

Although personality inventories had some success, mental health professionals 
working with psychiatric populations felt a need for additional help in diagnosing 
and treating mental illness. In the 1920s, a new gente of tools for the assessment 
of personality and psychopathology emerged. These instruments, known as pro- 
Jective techniques, had their roots in the free association methods pioneered by Gal- 
ton and used clinically by Kraepelin, Jung, and Freud. In 1921, a Swiss psychia- 
trist named Hermann Rorschach published a test consisting of ten inkblots to be 
presented for interpretation, one at a time, to the examinee. The key to the suc- 
cess of this first formal projective technique was that it provided a standardized 
method for obtaining and interpreting subjects’ responses to the inkblot cards, 
responses that—by and large—treflect the subject’s unique modes of perceiving 
and relating to the world. Rorschach’s test was taken up by several American psy- 
chologists and propagated in various universities and clinics in the United States 
after his untimely death in 1922. The Rorschach technique, along with other pic- 
torial, verbal, and drawing instruments, like the Thematic Apperception Test, 
sentence completion tests, and human figure drawings provided a whole new 
repertoire of tools—more subtle and incisive than the questionnaites—with 
which clinicians could investigate aspects of personality that test takers them- 
selves may have been unable or unwilling to reveal. Though there is much con- 
troversy about their validity, primarily because they often rely on qualitative in- 
terpretations as much as or more than on numerical scores, projective techniques 
are still a significant part of the toolkit of many clinicians (Viglione & Rivera, 
2003). 


Neuropsychological Tests 

The role of brain dysfunction in emotional, cognitive, and behavioral disorders 
has been increasingly recognized throughout the past century. However, the ma- 
jor impetus for the scientific and clinical study of brain-behavior relationships, 
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which is the subject of neuropsychology, came from Kurt Goldstein’s investigations 
of the difficulties he observed in soldiers who had sustained brain injuries during 
World War I. Often these soldiers showed a pattern of deficits involving prob- 
lems with abstract thinking, memory, as well as the planning and execution of rel- 
atively simple tasks, all of which came to be known under the rubric of organicity, 
which was used as a synonym for brain damage. Over several decades, a number 
of instruments meant to detect organicity, and distinguish it from other psychi- 
atric disorders, came into being. Many of these were variations of the perfor- 
mance—as opposed to verbal—tests that had been developed to assess general 
intellectual ability in individuals who could not be examined in English or who 
had hearing or speech impairments. These tests involved materials like form 
boards, jigsaw puzzles, and blocks as well as paper-and-pencil tasks such as mazes 
and drawings. A great deal has been learned about the brain and its functioning 
in the past few decades and much of the initial thinking in neuropsychological as- 
sessment has had to be revised based on new information. Brain damage is no 
longer viewed as an all-or-none condition of organicity with a common set of 
symptoms, but rather as a huge range of possible disorders resulting from the in- 
teraction of specific genetic and environmental factors in each individual case. 
Nevertheless, the field of neuropsychological assessment has continued to grow 
in the number and types of instruments available and has contributed both to 
the clinical and scientific understanding of the many and varied relationships 
between brain functioning and cog- 

nition, emotions, and behaviors 


(Lezak, 1995). = Ngpid Reference 15 
CURRENT USES OF Current Uses of 
PSYCHOLOGICAL TESTS Psychological Tests 


e The first and foremost use of tests 


Present-day testing is, on the whole, is in the pragmatic process of mak- 


more methodologically sophisticated ing decisions about people, either as 
and better informed than at any time individuals or as groups. 
in the past. The current uses of tests, e The second use of tests in terms of 


frequency and longevity is in scien- 


which take place in a wide variety of A OInaCaIORE. 


contexts, may be classified into three nomena and individual differences. 
categories: (a) decision-making, (b) ¢ The most recent, and least devel- 
psychological research, and (c) self- oped, use of tests is in the thera- 


peutic process of promoting self- 
understanding and psychological 
adjustment. 


understanding and personal develop- 
ment. As can be gleaned from this list, 
presented in Rapid Reference 1.5, the 
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three kinds of uses differ vastly in their impact and in many other respects, and the 
first one of them is by far the most visible to the public. 


Decision Making 


The primary use of psychological tests is as decision-making tools. This particu- 
lar application of testing invariably involves value judgments on the part of one 
ot more decision makers who need to determine the bases upon which to select, 
place, classify, diagnose, or otherwise deal with individuals, groups, organiza- 
tions, or programs. Naturally, this use of testing is often fraught with controversy 
since it often results in consequences that ate unfavorable for one ot mote par- 
ties. In many situations in which tests are used to make decisions and people dis- 
agree with the decisions made, the use of tests itself is attacked regardless of 
whether or not it was appropriate. 

When tests are used for making significant decisions about individuals or pro- 
grams, testing should be merely a part of a thorough and well-planned decision- 
making strategy that takes into account the particular context in which the deci- 
sions ate made, the limitations of the tests, and other sources of data in addition 
to tests. Unfortunately, very often—for reasons of expediency, carelessness, or 
lack of information—tests are made to bear the responsibility for flawed deci- 
sion-making processes that place too much weight on test results and neglect 
other pertinent information. A number of decisions made by educational, gov- 
ernmental, or corporate institutions on a routine basis, usually involving the si- 
multaneous evaluation of several people at once, have been and still are made in 
this fashion. Although they carry important consequences—such as employ- 
ment, admission to colleges or professional schools, graduation, or licensure to 
practice a profession—for the individuals involved, decisions are based almost 
exclusively on test scores. This practice, a legacy of the way in which testing orig- 
inated, is one that testing professionals, as well as some government agencies, are 
trying to change. One of several important steps in this direction is the publica- 
tion of a resource guide for educators and policymakers on the use of tests as part 
of high-stakes decision making for students (U.S. Department of Education, Of- 
fice for Civil Rights, 2000). 


Psychological Research 


Tests are often used in research in the fields of differential, developmental, ab- 
normal, educational, social, and vocational psychology, among others. They pro- 
vide a well-recognized method of studying the nature, development, and inter- 
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relationships of cognitive, affective, and behavioral traits. In fact, although a 
number of tests that originated in the course of psychological investigations have 
become commercially available, many more instruments remain archived in dis- 
sertations, journals, and various compendiums of experimental measures dis- 
cussed in Sources of Information about Tests at the end of this chapter. Because 
there are seldom any immediate practical consequences attendant to the use of 
tests in research, their use in this context is less contentious than when they are 
used in decision making about individuals, groups, organizations, or programs. 


Self-Understanding and Personal Development 


Most humanistic psychologists and counselors have traditionally perceived the 
field of testing, often justifiably, as overemphasizing the labeling and categoriza- 
tion of individuals in terms of rigid numerical criteria. Starting in the 1970s, a few 
of them, notably Constance Fischer (1985/1994), began to use tests and other as- 
sessment tools in an individualized manner, consonant with humanistic and exis- 
tential-phenomenological principles. This practice, which views testing as a way 
to provide clients with information to promote self-understanding and positive 
growth, has evolved into the “herapeutic model of assessment espoused by Finn and 
Tonsager (1997). Obviously, the most pertinent application of this model is in 
counseling and psychotherapeutic settings in which the client is the main and only 
user of test results. 


PSYCHOLOGICAL ASSESSMENT VERSUS 
PSYCHOLOGICAL TESTING 


For reasons that are mostly related to the marketing of tests, some test authors and 
publishers have begun to use the word assessment in the titles of their tests. Thus, in 
the mind of the general public the terms assessment and ‘esting are often seen as syn- 
onymous. This is an unfortunate development. The distinction between these 
terms is one that many people in the field believe is worth preserving, and one that 
the general public, as potential assess- 

ment clients or consumers of tests, 


should be aware of as well. DON’T FORGET 


The use of tests for making decj- — srrestesstssesssssnsccesectsnscrsscessonsscsssssenscrssernsconscsnssoressetssoene 
sions about a person, a group, or a e Tests and assessments are NOT syn- 
> ; onymous. 
program should always take place : 
e Tests are among the tools used in 


within the context of psychological as- the process of assessment. 


sessment. This process can occur in 


24 ESSENTIALS OF PSYCHOLOGICAL TESTING ) 


health care, counseling, or forensic settings, as well as in educational and em- 
ployment settings. Psychological assessment is a flexible, not standardized, process 
aimed at reaching a defensible determination concerning one or more psycho- 
logical issues or questions, through the collection, evaluation, and analysis of data 
appropriate to the purpose at hand (Maloney & Ward, 1976). 


Steps in the Assessment Process 


The first and most important step in psychological assessment is to identify its 
goals as clearly and realistically as possible. Without clearly defined objectives 
that are agreed upon by the assessor and the person requesting the assessment, 
the process is not likely to be satisfactory. In most instances, the process of as- 
sessment ends with a verbal or written report, communicating the conclusions 
that have been reached to the persons who requested the assessment, in a com- 
prtehensible and useful manner. In between these two points, the professional 
conducting the assessment, usually a psychologist or a counselor, will need to em- 
ploy her or his expertise at every step. These steps involve the appropriate selec- 
tion of instruments to be used in gathering data, their careful administration, 
scoring, interpretation, and—most important of all—the judicious use of the 
data collected to make inferences about the question at hand. This last step goes 
beyond psychometric expertise and requires a knowledge of the field to which the 
question refers, such as health care, educational placement, psychopathology, 
organizational behavior, or criminology, among others. Examples of issues 
amenable to investigation through psychological assessment include 


* diagnostic questions, such as differentiating between depression and de- 
mentia; 

* making predictions, such as estimating the likelihood of suicidal or homi- 
cidal behaviors; and 

* evaluative judgments, such as those involved in child custody decisions or 
in assessing the effectiveness of programs or interventions. 


None of these complex issues can be resolved by means of test scores alone be- 
cause the same test score can have different meanings depending on the exami- 
nee and the context in which it was obtained. Furthermore, no single test score 
ot set of scores can capture all the aspects that need to be considered in resolving 
such issues. 

Psychological tests may be key components in psychological assessment, but 
the two differ fundamentally in important ways. Rapid Reference 1.6 lists several 
dimensions that differentiate psychological testing and assessment. Even though 
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= fid Reference 1.6 


Typical Differences between Psychological Testing 


and Assessment 
Basis Psychological Testing Psychological Assessment 
Degree of Simpler; involves one uniform More complex; each assessment 
complexity procedure, frequently unidi- involves various procedures (in- 
mensional. terviewing, observation, test- 
ing, etc.) and dimensions. 
Duration Shorter, lasting from a few Longer, lasting from a few hours 
minutes to a few hours. to a few days or more. 
Sources of One person, the test taker. Often collateral sources, such 
data as relatives or teachers, are 
used in addition to the subject 
of the assessment. 
Focus How one person or group The uniqueness of a given indi- 


Qualifications 
for use 


Procedural 
basis 


Cost 


Purpose 


Degree of 
structure 


Evaluation of 
results 


compares with others 
(nomothetic). 


Knowledge of tests and testing 
procedures. 


Objectivity required; quantifica- 
tion is critical. 


nexpensive, especially when 
testing is done in groups. 


Obtaining data for use in 
making decisions. 


Highly structured. 


Relatively simple investigation 
of reliability and validity based 
on group results. 


vidual, group, or situation 
(idiographic). 
Knowledge of testing and other 
assessment methods as well as 
of the area assessed (e.g., psy- 
chiatric disorders, job require- 
ments). 


Subjectivity, in the form of clin- 
ical judgment, required; quan- 
tification rarely possible. 


Very expensive; requires inten- 
sive use of highly qualified pro- 
fessionals. 


Arriving at a decision concern- 
ing the referral question or 
problem. 


Entails both structured and 
unstructured aspects. 
Very difficult due to variability 


of methods, assessors, nature 
of presenting questions, etc. 
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there is little question about the general superiority of assessment over testing 
with regard to comprehensiveness and utility, the greater complexity of the as- 
sessment process makes its results far more difficult to evaluate than those of 
testing. Nevertheless, in recent years, evidence of the efficacy of assessment, at 
least in the realm of health care delivery, has begun to be assembled (Eisman et 
al., 2000; Kubiszyn et al., 2000; Meyer et al., 2001). 


TEST USER QUALIFICATIONS 


As the number of tests has continued to grow and their uses have expanded, not 
onlyin the United States but around the world, the question of test misuse has be- 
come of increasing concern for the public, the government, and various profes- 
sions. Psychology, which is the profession from which tests arose and the one 
with which they are most distinctly associated, has taken the lead in trying to com- 
bat their misuse. The Zesting Standards promulgated by the APA and other profes- 
sional organizations (AERA, APA, & NCME, 1999) are a major vehicle to this 
end. The APA also addresses issues related to testing and assessment in its ethi- 
cal principles and code of conduct (APA, 2002), as do other professional associ- 
ations (e.g, American Counseling Association, 1995; National Association of 
School Psychologists, 2000). 

Although the technical qualities of a number of tests are far from ideal and can 
contribute to problems in their use, it is generally conceded that the primary rea- 
son for test misuse lies in the insufficient knowledge or competence on the part 
of many test users. Tests may appear relatively simple and straightforward to po- 
tential users who are unaware of the cautions that must be exercised in their ap- 
plication. Because of this, in the past few decades, professional associations in the 
United States and elsewhere have been developing documents that outline more 
clearly and specifically than ever before the skills and knowledge base required 
for competent test use (American Association for Counseling and Development, 
1988; Eyde, Moreland, Robertson, Primoff, & Most, 1988; International Test 
Commission, 2000; Joint Committee on Testing Practices, 1988). 

One of the clearest expositions of these requirements is in a report prepared 
over the course of five years by the APA Task Force on Test User Qualifications 
(APA, 2000). This report outlines (a) the core knowledge and skills essential to 
those who use tests to make decisions or formulate policies that affect the lives 
of test takers, and (b) the expertise that test users in the specific contexts of em- 
ployment, education, career counseling, health care, and forensic work must pos- 
sess. Core or generic knowledge and skills in psychometrics, statistics, test selec- 
tion, administration, scoring, reporting, and safeguarding are considered relevant 
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to all test users. Additional knowledge and supervised experience required for the 
use of tests in the various contexts and with diverse groups of test takers are also 
outlined in the report, as are the variety of uses of tests in classification, descrip- 
tion, prediction, intervention planning, and tracking in each context. 

Another aspect of testing that has contributed to test misuse over the decades 
is the relative ease with which test instruments can be obtained by people who ate 
not qualified to use them. To some extent, the availability of tests is a function of 
the freedom with which information flows in democratic societies like the United 
States, especially in the era of the World Wide Web. Another reason for this prob- 
lem—alluded to earlier in this chapter—is the fact that many tests are commer- 
cial products. As a result, some test publishers have been willing to sell tests to 
persons or institutions without using adequate safeguards to ascertain whether 
they possess the proper credentials. At one point, during the 1950s and 1960s, the 
Testing Standards included a three-tiered system for classifying tests in terms of the 
qualifications needed for their use (APA, 1966, pp. 10—11). This system, which la- 
beled tests as Level A, B, or C depending on the training required to use them, was 
easily circumvented by individuals in schools, government agencies, and busi- 
nesses. Although many test publishers still use the system, the Testing Standards no 
longer do. Rapid Reference 1.7 outlines the elements typically included in a three- 
tiered classification system of test user qualifications. 

In 1992, a number of the publishers of tests and providers of assessment ser- 
vices established the Association of Test Publishers (ATP). This nonprofit orga- 
nization tries to uphold a high level of professionalism and ethics in the testing 
enterprise. One way in which they monitor the distribution of tests is by requir- 
ing some documentation attesting to a minimum level of training from those who 
would purchase their products. 

Qualification forms for test purchase are now included in the catalogs of all 
reputable test publishers. No matter how sincere publishers may be in their ef- 
forts to preserve the security of test materials and to prevent their misuse, the ef- 
fectiveness of these efforts is by necessity limited. Not only is it not feasible to 
verify the qualifications that purchasers claim on the forms they submit, but in 
addition no formal set of qualifications—whether by education or by licensure— 
can ensure that an individual is competent to use a particular test properly in a 
given situation (see Chapter 7). 


SOURCES OF INFORMATION ABOUT TESTS 


In psychological testing, as in every other human endeavor, the Internet has cre- 
ated an inexhaustible supply of information. Thus, alongside the print references 
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that the field has traditionally had, there now is a large number of on-line and elec- 
tronic media resources that are easily accessible. 


Internet Resources 


For the person who seeks information about psychological tests, a good starting 
point is the Testing and Assessment section of the APA’s Web site (http://www 
.apa.org). Within this section, among other things, there is an excellent article on 
“PAQ/Finding Information About Psychological Tests” (APA, 2003) that pro- 
vides guidance on how to locate published and unpublished tests as well as im- 
portant documents relevant to psychological testing. Published tests are commer- 
cially available through a test publisher, although they sometimes go out of print 
as books do. Unpublished tests have to be obtained directly from the individual in- 
vestigator who created them, unless they appear in the periodical literature or in 
specialized directories (discussed shortly). 

Two other great entry points on the Internet, for those who seek information 
about a specific test, are (a) the Buros Institute of Mental Measurements (BI) Test 
Reviews Online Web page (http://www.unl.edu/buros), which offers free in- 
formation on nearly 4,000 commercially available tests as well as more than 2,000 
test reviews that can be purchased and displayed online; and (b) the Educational 
Testing Service (ETS) Test Collection database (at http://www.ets.org/testcoll/ 
index.html), which is the largest of its kind in the world. In addition, the Educa- 
tional Resources Information Center (ERIC) system Web site (http://eric.ed 
.gov)— funded by the U.S. Department of Education—contains a wealth of ma- 
terials related to psychological testing, 

Another way to obtain informa- 
tion about both published and un- 
published tests online is through the DON’T FORGET 
electronic indexes of the periodical E eet T A TN eee ee 
literature in psychology, education, or One of the most basic distinctions 
business. The PsycINFO database of aimon wesie comecen whether wey 


he APA ‘lable th a li are published. 
nes AVARAS ONEAN T e Published tests are commercially 


braries or by subscription, provides available through test publishers. 
an entry point at which to use the e Unpublished tests must be obtained 
name of a test to find bibliographic from the individual investigator who 
references, abstracts, and even full developed them, from special di- 


rectories of unpublished measures, 


text of articles about it. In addition to or from the periodical literature. 


exact titles, PsycINFO and other 
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DON’T FORGET 


Appendix A lists all of the commer- 
cially available, published tests and 
psychological assessment instruments 
mentioned throughout this book, 
along with codes identifying their pub- 
lishers. 


Appendix B provides current Internet 
addresses for the publishers listed in 
Appendix A. More detailed informa- 
tion on test publishers, including street 
addresses and telephone numbers, is 
available in the latest edition of Tests in 
Print (Murphy, Plake, Impara, & Spies, 
2002). 


databases also can be searched by sub- 
jects, keywords, and authors, which 
makes them especially useful when 
only partial information is available. 

Once a test is located through any 
of these resources, one can usually 
also determine whether it is pub- 
lished and how it can be obtained. If 
the test is published, it may be 
ordered from the company that pub- 
lishes it by those who meet the quali- 
fications to use it. Ordering informa- 
tion is available in the publishers’ 
catalogs, many of which are now 
available online as well as in printed 


form. The ATP Web site (http://www.testpublishers.org) has links to many test 
publishers and providers of assessment services. Internet addresses for all of the 


organizations mentioned in this section, and other important sources of infor- 


mation on tests, can be found in Rapid Reference 1.8. 


= fid Reference 1.8 


Internet Sources of Information on Psychological Tests 


Organization (Acronym) 


Website 


American Educational Research 
Association (AERA) 


American Psychological Association 
(APA) 
Association of Test Publishers (ATP) 


Buros Institute of Mental Measurements 


Educational Resources Information 
Center (ERIC) 


Educational Testing Service (ETS) 
International Test Commission (ITC) 


National Council on Measurement in 
Education (NCME) 


http://www.aera.net 
http://www.apa.org 


http://www.testpublishers.org 
http://www.unl.edu/buros 


http://eric.ed.gov 


http://www.ets.org/testcoll/index.htm| 
http://www.intestcom.org 


http://www.ncme.org 
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Print Resources 


Published Tests 

As far as commercially available, published tests are concerned, the most impor- 
tant sources of information stem from the Buros Institute of Mental Measure- 
ments (BI) in Lincoln, Nebraska. In particular, the BI (http://www.unl.edu/ 
buros) produces two series of volumes that can guide users to almost every pub- 
lished test available in the United States. One of these is the Tests in Print (TIP) se- 
ties and the other is the Mental Measurements Yearbook (MMY ) series. Tests in Print 
is acomprehensive bibliography of all tests that are commercially available at the 
time a given volume of the series is published. Each entry has the test title, 
acronym, author, publisher, publication date, and other basic information about 
the test as well as cross-references to the reviews of the test in all the MMYs avail- 
able at that point. In addition, the 77P series contains an extremely useful classi- 
fied index of tests that are in print, as well as indexes of test scores, publishers, 
acronyms, and names of authors and reviewers. The MMY series, in turn, goes 
back to 1938, when the late Oscar Buros published the first yearbook to assist test 
users by providing evaluative test reviews written by qualified and independent 
professionals. Although the MMYs are still published in book form, their entries 
and reviews are also available online and in other electronic media. The Buros In- 
stitute also publishes many other test-related materials. 

PRO-ED (http://www.ptoedinc.com) is the publisher of Zés¢s, a series of en- 
cyclopedic volumes listing short descriptions of instruments in psychology, edu- 
cation, and business. The Test Critiques series, dating back to 1984, is the compan- 
ion to Zests. Each volume in this series contains test reviews and cumulative 
indexes to all its previous volumes. 


Unpublished Tests 

The goal of behavioral scientists who use psychological tests is to investigate psy- 
chological constructs as well as individual and group differences. Many existing 
tests are used exclusively for scientific research and are not commercially avail- 
able. These tests are referred to as unpublished measures because they cannot be 
purchased; conditions for their use are typically established by the authors of each 
instrument and most often require a letter requesting permission to use them. In- 
formation about unpublished tests—and often the instruments themselves—is 
available in the periodical literature in psychology (e.g., through PsycINFO on- 
line) and in various directories (e.g., Goldman, Mitchell, & Egelson, 1997; Robin- 
son, Shaver, & Wrightsman, 1991). The previously mentioned article “FAQ/ 
Finding Information About Psychological Tests” (APA, 2003) lists several print 
and electronic resources for information on unpublished tests. 
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‘a TEST YOURSELF wA. 


|. Which of the following is not an essential element of psychological 
testing? 


a) Systematic procedures 

b) The use of empirically derived standards 
c) Preestablished rules for scoring 

d) Sampling behavior from affective domains 


2. The single most important source of criteria for evaluating tests, testing 
practices, and the effects of test use can be found in the 


a) Ethical Principles of Psychologists and Code of Conduct. 
b) Standards for Educational and Psychological Testing. 
c) Diagnostic and Statistical Manual of Mental Disorders. 
d) Report of the Task Force on Test User Qualifications. 


3. The earliest antecedents of modern testing for personnel selection date 
back to 


a) China, B.C.E. 

b) ancient Greece. 

c) the Inca empire. 

d) Medieval Europe. 

4. Evaluating psychological tests is least problematic 
a) prior to their being placed into use. 

once they have been placed into use. 


joy 


5. Compared to the other areas listed, the development of criteria or bases 
for decision making has been substantially slower in the context of 


a) educational assessment. 
b) occupational assessment. 
c) clinical assessment. 


6. Credit for devising the first successful psychological test in the modern 
era is usually given to 


a) Francis Galton. 

b) Alfred Binet. 

c) James McKeen Cattell. 
d) Wilhelm Wundt. 
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7. The true ratio IQ or intelligence quotient was derived by 


a) adding the mental age (MA) and the chronological age (CA) of the test 
taker. 


b) subtracting the CA from the MA and multiplying the result by |00. 
dividing the CA by the MA and multiplying the result by 100. 
) dividing the MA by the CA and multiplying the result by | 00. 


8. The primary purpose for which psychological tests are currently used is 


au £0) 


a) psychological research. 


b) educational research. 


c) decision making. 

d) self-understanding and personal development. 

9. Compared to psychological testing, psychological assessment is generally 
a) simpler 

b) more structured. 

c) more expensive. 

d) more objective. 


10. Which of the following would be the best source of information on a test 
that is not commercially available? 


a) Mental Measurements Yearbooks 
b) Test Critiques 

c) Tests in Print 

d) PsycINFO 


Answers: |. d; 2. b; 3.a;4.a;5.¢; 6.b; 7.d;8.¢9.c 10.d. 


Two 


ESSENTIAL STATISTICS FOR TESTING 


y and large, the progress of science dovetails with the invention of mea- 

suring tools and advances in measurement procedures and techniques. 

The science of astronomy, for example, really took offin the 17th and 18th 
centuries following the invention of a telescope suitable for observing the cos- 
mos and Descartes’s invention of analytic geometry, which led to a mote precise 
calculation of distances between celestial bodies, among other things. Similarly, 
the enormous current strides in the field of neuroscience owe much to the devel- 
opment of techniques such as positron emission tomography (PET) and func- 
tional magnetic resonance imaging (fMRI), which allow scientists to visualize and 
measure small biochemical changes and events in the brain. 

As we saw in Chapter 1, the modern field of psychological testing also had its 
start with the invention of successful tools. The Binet-Simon intelligence scales 
provided for the measurement of important cognitive processes—such as com- 
prehension, judgment, and memory—through behavior samples calibrated ac- 
cording to age. Arthur Otis’s invention of objective multiple choice items led to 
the first group tests of general intelligence. Statistical techniques developed at ap- 
proximately the same time as the first tests allowed for the analysis of data col- 
lected by means of those tests. 


MEASUREMENT 


The concept of measurement is at the heart of psychological testing as a scientific 
enterprise for the study of human behavior. Measurement involves the use of cer- 
tain devices or rules for assigning numbers to objects or events (Stevens, 1946). If 
we apply this process systematically, then to a large extent, a phenomenon that is 
measured is made more easily subject to confirmation and analysis, and thus is 
made mote objective as well. In other words, by systematically analyzing, catego- 
rizing, and quantifying observable phenomena we place them in the scientific arena. 

Central to the definition of psychological tests is the fact that they consist of 
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catefully chosen samples of behavior to which a numerical or category system is 
applied according to some preestablished standards. Psychological testing is 
largely coextensive with the field of psychometrics, or psychological measurement, 
and is one of the primary tools for the science and practice of psychology. 

The use of numbers in testing requires us to delve into statistics. For many stu- 
dents of psychology the use of statistics and quantitative data in general poses a 
problem that may seem insurmountable: namely, that dealing with numbers tends 
to cause some anxiety. This anxiety is connected with the distress that classes in 
mathematics and statistics often induce for reasons that may be related as much 
to emotional or attitudinal factors as to those subjects themselves or to the way 
they have been taught traditionally. This chapter presents the statistical concepts 
needed to understand the basic principles of psychological testing. Those who 
have mastered basic statistics may be able to skip all or most of the chapter. As 
for the rest, any motivated reader of this book can achieve a serviceable grasp of 
the concepts described here. It is important, however, to realize that these con- 
cepts follow a logical progression; in order to proceed to each new topic it is es- 
sential to master the preceding ones. Additional help in understanding basic sta- 
tistical methods is readily available in many excellent textbooks, such as the ones 
listed in Rapid Reference 2.1. 


VARIABLES AND CONSTANTS 


One of the most basic distinctions we can make in any science is that between 
variables and constants. As the terms themselves imply, a variable is anything that 
varies whereas a constant is anything that does not. Our world has many variables 
and few constants. One example of a constant is T (pi), the ratio of the circum- 
ference of a circle to its diameter, a number that is usually rounded to 3.1416. 
Variables, on the other hand, are everywhere and they can be classified in a mul- 
titude of ways. For example, some variables are visible (e.g., sex, color of eyes) and 
others invisible (e.g., personality, intelligence); some are defined so as to pertain 
to very small sets and others to very large sets (e.g., the number of children in a 
family or the average income of individuals in a country); and some are continu- 
ous, others discrete. 

This last distinction is important for our purposes and bears some explaining. 
Technically, discrete variables are those with a finite range of values—or a poten- 
tially infinite, but countable, range of values. Dichotomous variables, for instance, 
are discrete variables that can assume only two values, such as sex or the outcome 
of coin tosses. Po/ytomous variables are discrete variables that can assume more 
than two values, such as marital status, race, and so on. Other discrete variables 
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Advice on Statistics 


Basic Premises 


|. To understand psychological tests, one needs to deal with numbers and statis- 
tics. 


2. Understanding statistics is possible for anyone who reads this book. 
3. The best way to increase one's grasp of statistical concepts is to apply them. 


Recommended Sources of Help with Statistics 
Books 


e Howell, D. C. (2002). Statistical methods for psychology (5th ed.). Pacific Grove, 
CA: Duxbury. 

e Kirk, R. E. (1999). Statistics: An introduction (4th ed.). Fort Worth, TX: Harcourt 
Brace. 

e Urdan, T. C. (2001). Statistics in plain English. Mahwah, NJ: Erlbaum. 

e Vogt, W. P (1998). Dictionary of statistics and methodology: A nontechnical guide 
for the social sciences (2nd ed.). Thousand Oaks, CA: Sage. 

Video 

e Blatt, |. (Producer/Writer/Director). (1989). Against all odds: Inside statistics 
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can assume a wider range of values but can still be counted as separate units; ex- 
amples of these are family size, vehicular traffic counts, and baseball scores. Al- 
though in practice it is possible to make errors in counting, in principle, discrete 
variables can be tallied precisely and without error. 

Continuous variables such as time, distance, and temperature, on the other hand, 
have infinite ranges and really cannot be counted. They are measured with scales 
that could be theoretically subdivided into infinity and have no breaks in between 
their points, such as the scales in analog clocks, yardsticks, and glass thermometers. 
Since our measuring instruments (even atomic clocks!) can never be calibrated 
with enough precision to measure continuous variables exactly, the measurements 
we take of such variables are more or less accurate approximations. 

Before we start dealing with numbers, another word of caution is in order. In 
psychological testing, we are almost always interested in variables that are con- 
tinuous (e.g., degrees of integrity, extraversion, or anxiety), yet we measure with 
tools, such as tests or inventories, that are not nearly as precise as those in the 
physical and biological sciences. Even in those sciences the discrete measutement 
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of continuous variables poses some 
limitations on the accuracy measure- 
ments. It therefore stands to reason 
that in the behavioral sciences we 
must be particularly aware of poten- 
tial sources of error and look for per- 
tinent estimates of error whenever 
we are presented with the results of 
any measurement process. For ex- 
ample, if polls taken from samples of 
potential voters are used to estimate 
the outcome of an election, the esti- 
mated margins of error have to be 
displayed alongside the results of the 
polls. 

In summary, when we look at the 
results of any measutement process, 


DON’T FORGET 


e Although numbers may seem pre- 
cise, all measurements are prone to 
error. 


e When we are measuring discrete 
variables, errors arise only from in- 
accurate counting. Good practice 
requires the prevention, detection, 
and correction of inaccurate count- 

ing. 

e When we are measuring continu- 

ous variables, on the other hand, 

measurement error is inevitable, as 

a consequence of the limitations of 

measurement tools. 


e As measurement tools, psychologi- 
cal tests are subject to many limita- 
tions. Hence, margins of error must 
always be estimated and communi- 


we need to hold clearly in mind the cated along with test results. 

fact that they ate inexact. With regard 

to psychological testing in particular, 

whenever scores on a test are reported, the fact that they are estimates should be 
made clear; furthermore, the limits within which the scores might range as well as 
the confidence levels for those limits need to be given, along with interpretive in- 


formation (see Chapter 4). 


THE MEANING OF NUMBERS 


Because numbers can be used in a multitude of ways, S. S. Stevens (1946) devised 
a system for classifying different levels of measurement on the basis of the rela- 
tionships between numbers and the objects or events to which the numbers are 
applied. These levels of measurement or scales—outlined in Table 2.1—specify 
some of the major differences in the ways numbers may be used as well as the 
types of statistical operations that are logically feasible depending on how num- 
bers are used. 


Nominal Scales 


At the simplest level of his classification, Stevens placed what he called nominal 
scales. The word nominal is derived from the Latin root nomen, meaning name. As 
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this implies, in such scales, numbers are used solely as labels to identify an indi- 
vidual or a class. The nominal use of numbers to label individuals is exemplified 
by the Social Security numbers (SS#s) that identify most people who live in the 
United States; these numbers are useful because each is assigned to only one per- 
son and can therefore serve to identify persons more specifically than their first 
and last names, which can be shared by many people. Numbers can also be used 
to label categorical data, which are data related to variables such as gender, political 
affiliation, color, and so forth—that is, data that derive from assigning people, 
objects, or events to particular categories or classes. When entering demographic 
data into a computer for analysis, for instance, investigators typically create a 
nominal scale that uses numbers to indicate the levels of a categorical variable. 
For example, the number 1 (one) may be assigned to all females and 2 (two) to all 
males. The only requirement for this use of numbers is that all the members of a 
set designated by a given number should be equal with regard to the category as- 
signed to that number. Naturally, while the numbers used in nominal scales can 
certainly be added, subtracted, multiplied, or divided, the results of such opera- 
tions are not meaningful. When we use numbers to identify categories, such as 
pass-fail or psychiatric diagnoses, the only property of such numbers is identity; 
this means that all members of a category must be assigned the same number and 
that no two categories may share the same number. The only permissible arith- 
metic operation is counting the frequencies within each category. One can then, 
of course, manipulate those frequencies further by calculating proportions and 
doing further analyses based on them. 


Ordinal Scales 


The numbers used in ordinal scales convey one more bit of meaning than those in 
nominal scales, albeit a significant one. In these scales, in addition to identity, 
there is the property of rank order, which means that the elements in a set can be 
lined up in a series— from lowest to highest or vice versa—arranged on the ba- 
sis of a single variable, such as birth order or level of academic performance 
within a given graduating class. Although rank order numbers convey a precise 
meaning in terms of position, they carry no information with regard to the dis- 
tance between positions. Thus, the students in a class can be ranked in terms of 
their performance, but this ranking will not reflect the amount of difference be- 
tween them, which could be great or small. Similarly, in any hierarchical organi- 
zation, say, the U.S. Navy, ranks (e.g., ensign, lieutenant, commander, captain, 
admiral) denote different positions, from lowest to highest, but the differences 
between them in terms of accomplishments or prestige are not the same. If those 
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ranks were assigned numbers, such as 1, 3, 7, 14, and 35, the order of precedence 
would be maintained, but no further meaning would be added. 

In psychological testing, the use of ordinal numbers to convey test results is 
pervasive. Rank ordered test scores are reported as percentile rank (PR) scores—not 
to be confused with the familiar percentage scores widely used in school grading. 
Percentile scores are simply ordinal numbers set on a scale of 100, so that the rank 
indicates the percentage of individuals in a group who fall at or below a given level 
of performance. For example, the percentile rank score of 70 indicates a level of 
performance that equals or exceeds that of 70% of the people in the group in 
question. Percentile rank scores, often referred to simply as percentiles, are the main 
vehicle whereby test users convey normative information derived from tests, and 
thus they will be discussed again, at greater length, in the next chapter. 

Numerical data from ordinal scales can be manipulated statistically in the same 
way as nominal data. In addition, there are a few statistical techniques, such as 
Spearman’s tho (r,) correlation coefficient for rank differences, that are specifi- 
cally appropriate for use with ordinal data. 


Interval Scales 


In interval scales, also known as equa/-unit scales, numbers acquire yet one more 
important property. In these scales, the difference between any two consecutive 
numbers reflects an equal empirical or demonstrable difference between the ob- 
jects or events that the numbers represent. An example of this is the use of days 
to mark the passage of calendar time. One day consists of 24 hours, each hour of 
60 minutes, and each minute of 60 seconds; if two dates are 12 days apart, they are 
exactly three times as far apart as two dates that are only 4 days apart. Note, how- 
ever, that calendar time in months is not an equal-unit scale because some months 
are longer than others. Furthermore, calendar time also typifies a characteristic of 
interval scales that limits the meaning of the numbers used in them, namely, that 
there is no true zero point. In the case of calendar time, there is no agreed upon 
starting point for the beginning of time. Different cultures have devised arbitrary 
starting points, such as the year Christ was presumed to have been born, to mark 
the passage of years. For instance, the much anticipated arrival of the new mille- 
nium at the end of the year 2000 of the Christian or Common Era came in the 
year 5761 of the Jewish calendar and in the year 4699 of the Chinese calendar, 
both of which start many years before the beginning of the Common Era. 

In interval scales, the distances between numbers are meaningful. Thus, we 
can apply most arithmetical operations to those numbers and get results that 
make sense. However, because of the arbitrariness of the zero points, the num- 
bers in an interval scale cannot be interpreted in terms of ratios. 
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Ratio Scales 


Within ratio scales, numbers achieve the property of additivity, which means they 
can be added—as well as subtracted, multiplied, and divided—and the result ex- 
pressed as a ratio, all with meaningful results. Ratio scales have a true or absolute 
zero point that stands for “none of” whatever is being measured. In the physical 
sciences, the use of this type of measurement scale is common; times, distances, 
weights, and volumes can be expressed as ratios ina meaningful and logically con- 
sistent way. For instance, an object that weighs 16 pounds is twice as heavy as one 
that weighs 8 pounds (16/8 = 2), just as an 80-pound object is twice as heavy as 
a 40-pound object (80/40 = 2). In addition, the zero point in the scale of weights 
indicates absolute weightlessness. In psychology, ratio scales are used primarily 
when we measure in terms of frequency counts or of time intervals, both of which 
allow for the possibility of true zeros. 

Categorical or discrete data can be measured—or accounted for—only with 
nominal scales, or with ordinal scales if the data fall in a sequence of some kind. 
Continuous, or metric, data can be measured with interval scales, or ratio scales 
if there is a true zero point. In addition, continuous data can be converted into 
classes or categories and handled with nominal or ordinal scales. For instance, we 
could separate people into just three categories—tall, medium, and short—by 
establishing a couple of arbitrary cut- 
off points in the continuous variable 


of height. DON’T FORGET 


When we move from nominal to 
e In measurement there has to be a 


ratio scales, we go from numbers that Heese blaine enw aE 
catty less information to numbers numbers applied to objects, events, 
that carry more. As a consequence of or people and the reality the num- 


this, going from one level of mea- pets cel 


e When the rules used to create this 


surement to another requires us to be link are not understood, the results 


aware of whether the information of the measurement process are 
that the numbers entail is preserved easily misinterpreted. 
through whatever transformations or e As we shift from one level of mea- 


surement to another, we must be 
aware of whether the information 
the numbers entail is preserved in 
the transformations or manipula- 
Why Is the Meaning of tions we apply. 

Numbers Relevant to e Scores are numbers with specific 
meanings. Unless the limitations in 
the meaning of scores are under- 
stood, inaccurate inferences are 
ikely to be made from the scores. 


manipulations we apply to them. 


Psychological Testing? 


Though it is not universally favored, 
Stevens’s system for classifying scales 
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of measurement helps to keep the re/ativity in the meaning of numbers in proper 
perspective. The results of most psychological tests are expressed in scores, 
which are numbers that have specific meanings. Unless the limitations in the 
meaning of scores are understood, inaccurate inferences are likely to be made on 
the basis of those scores. Unfortunately, this is too often the case, as can be seen 
in the following examples. 

Example 1: Specific limitations of ordinal scales. As mentioned earlier, many scores 
are reported in the form of percentile ranks, which are ordinal-level numbers that 
do not imply equality of units. If two scores are separated by 5 percentile rank 
units—e.g., the 45th and 50th percentiles—the difference between them and 
what the difference represents in terms of what is being measured cannot be 
equated with the difference separating any other scores that are 5 percentile units 
apart—for example, the 90th and 95th percentiles. In a distribution of scores that 
approximates the normal curve, discussed later in this chapter and portrayed in 
Figure 2.2, the majority of test scores cluster around the center of the distribu- 
tion. This means that in such distributions differences between rank scores are al- 
ways greater at the extremes or tails of the distribution than they are in the middle. 

Example 2: The problem of ratio IQs. The original intelligence quotients devised 
for use with the Stanford-Binet Intelligence Scale (S-B) were ratio IQs. That is to 
say, they were real guotients, derived by dividing the mental age (MA) score a child 
had obtained on the S-B test by the child’s chronological age (CA) and multiply- 
ing the result by 100 to eliminate the decimals. The idea was that average children 
would have similar mental and chronological ages and IQs of approximately 100. 
Children functioning below the average would have lower mental than chrono- 
logical ages and IQs below 100, while those functioning above the average would 
have higher mental than chronological ages and IQs above 100. This notion 
worked fairly well for children in the early and middle school ages during which 
there tends to be a somewhat steady pace of intellectual growth from year to year. 
However, the MA/CA ratio simply did not work for adolescents and adults be- 
cause their intellectual development is far less uniform—and changes are often 
imperceptible—from year to year. The fact that the maximum chronological age 
used in calculating the ratio IQ of the original S-B was 16 years, regardless of the 
actual age of the person tested, created additional problems of interpretation. 
Furthermore, the mental age and chronological age scales are not at the same 
level of measurement. Mental age, as assessed through the first intelligence tests, 
was basically an ordinal-level measurement, whereas chronological age can be 
measured on a ratio scale. For these reasons, dividing one number by the other to 
obtain a quotient simply did not lead to logically consistent and meaningful re- 
sults. Rapid Reference 2.2 shows numerical examples highlighting some of the 
problems that have caused ratio IQs to be abandoned. 
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Examples of Ratio IQ Computation With Attendant Problems 


Mental Chronological 


Age Age Difference 
Subject (MA) (CA) (MA - CA) Ratio IQ? 
Ally 6 years 5 years | year 6/5 x 100 = 120 
Ben |2 years 10 years 2 years 12/10 x 100 = 120 
Carol 18 years 15 years 3 years 18/15 x 100 = 120 


aln the actual computation of ratio IQs, both mental ages and chronological ages were expressed 
in months instead of years. 


Problem |: The mental age score required to obtain any given IQ keeps rising for 
each successive chronological age, so that the ratio IQs at different chronological 
ages are not equivalent. 


Problem 2: Whereas chronological age rises steadily, mental age does not. Since 

the highest mental age achievable on a given intelligence test cannot be limitless, 
even when a limit is placed on the maximum chronological age used to compute 
IQs—as was done in the S-B scale for a long time—the IQs that most adults can 
attain are artificially constrained compared to those of children and adolescents. 


Solution: Because of this and other problems with ratio IQs, as well as with the 

concept of mental ages, the use of the ratio IQ has been abandoned. The term 
IQ is now used for a score that is not a ratio IQ and is not even a quotient. This 
score, known as the deviation IQ, was pioneered by David Wechsler and is ex- 
plained in Chapter 3. 


What Can We Conclude About the Meaning of Numbers 
in Psychological Measurements? 


In psychology, it is essential to keep in mind that most of our measurement scales 
are of an ordinal nature. Equality of units is approximated by the scales used in 
many types of test scores, but such equality is never as permanent or as complete 
as it is in the physical sciences, because the units themselves ate relative to the per- 
formance of the samples from which they are derived. The use of ratio scales in 
psychology is limited to measures of frequencies, reaction times, or variables that 
can be meaningfully expressed in physical units. For example, if we were using as- 
sembly-line output per hour as a measure of speed of performance in an assembly 
line job, we could say that Worker A, who produces 15 units per hour, is 3 times 
as fast as Worker B, who produces only 5 units per hour. Note, however, that we 
could not say that Worker A is 3 times as good an employee as Worker B, because 
speed is probably not the only index of job performance even in an assembly line 
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operation. Overall level of performance is a more complex variable that most 
likely can be assessed only with a qualitative, ordinal scale. 


TYPES OF STATISTICS 


Since the use of numbers to represent objects and events is so pervasive in psy- 
chological testing, the field involves substantial application of statistics, a branch of 
mathematics dedicated to organizing, depicting, summarizing, analyzing, and oth- 
erwise dealing with numerical data. Numbers and graphs used to describe, con- 
dense, or represent data belong in the realm of descriptive statistics. On the other hand, 
when data are used to estimate population values based on sample values or to test 
hypotheses, cnferential statistics—a more ample set of procedures based on proba- 
bility theory—are applied. Fortunately, although both descriptive and inferential 
statistics are used extensively in the development of tests, most of the quantitative 
aspects of test score interpretation require only a good grasp of descriptive statis- 
tics and a relatively small number of techniques of the inferential type. Moreover, 
even though a background in higher level math is desirable in order to understand 
thoroughly the statistics involved in testing, it is possible to understand them at a 
basic level with a good dose of logic and a relatively limited knowledge of math. 
The words statistic and statistics are also used to refer to measures derived from 
sample data—as opposed to those derived from populations, which are called pa- 
rameters. Means, standard deviations, correlation coefficients, and other such 
numbers calculated from sample data are all statistics derived in order to estimate 
what is of real interest, namely, the respective population parameters. Parameters 
are mathematically exact numbers (or constants, such as T) that are not usually at- 
tainable unless a population is so fixed and circumscribed that all of its members 
can be accounted for, such as all the 


DON’T FORGET members of a college class in a given 


semestet. In fact, one of the main 


The Two Meanings of Statistics purposes of inferential statistics is to 
|. The study and application of meth- estimate population parameters on 
ods for organizing, depicting, sum- the bases of sample data and proba- 


marizing, analyzing, and otherwise AF 

Seine ata ee data. bility theory: 

2. Numbers (e.g., means, correlation 
coefficients) that describe the char- 
acteristics of variables or of sets of 
data derived from samples, as op- 
posed to those derived from popu- 
lations, which are called parameters. 


Descriptive Statistics 


Raw data are unwieldy. They usually 
consist of a bunch of numbers that 
do not convey much meaning, even 
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Table 2.2 Raw Data: 60 Test Scores 


41 50 39 40 40 31 42 29 37 36 
35 45 4 9 38 34 35 32 1 1 
39 47 30 45 43 47 35 46 
34 37 38 40 39 39 36 32 48 39 
33 42 44 48 47 40 33 46 
44 37 45 43 39 42 37 45 43 38 


after close examination, such as the 60 numbers listed in Table 2.2. These num- 
bers are the scores of 60 college students on the first test given in a psychological 
testing course; the test consists of 50 multiple-choice items. Simply looking at the 
numbers in the table yields some information, such as the fact that most scores 
seem to be somewhere between 30 and 50. With descriptive statistics, we can 
summarize the data so they are easier to understand. One way to summarize data 
is to represent them graphically; another way is to condense them into statistics 
that represent the information in a data set numerically. 


Frequency Distributions 

Before applying any statistical formulas, it is always a good idea to organize raw 
data in some sensible way so they can be inspected. Normally, this is accom- 
plished by means of a frequency distribution. Table 2.3 presents a distribution of the 
test scores in Table 2.2, listing the number of times or frequencies with which 
each score occurred and the percentage of times that it occurred. The Cumula- 
tive Percent column shows the consecutive addition of the numbers in the Per- 
cent column from the lowest to the highest scores. This last set of figures allows 
us to see the percentage of the 60 cases that fell at or below each score and can 
thus be easily read as percentile rank scores for the individuals who obtained each 
score. 

When the range of scores is very great, grouped frequency distributions help to or- 
ganize scores into a still more compact form. In these distributions, scores are 
grouped into intervals of a convenient size to accommodate the data, and the fre- 
quencies are listed for each interval instead of for each of the scores. Naturally, 
what is gained in compactness is lost in terms of the detail of information. 


Graphs 

Once the data have been organized into a frequency distribution, they can be trans- 
posed into any one of several graphic formats, such as pie charts or bar graphs (for 
discrete or categorical data) and histograms or frequency polygons (for continu- 
ous of metric data). The data from Table 2.3 are graphically displayed in the form 
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Table 2.3 Frequency Distribution of 60 Test Scores 


Scores Frequency (f) Percent (P) Cumulative Percent* (CP) 
50 1 1.7 100.0 
49 1 1.7 98.3 
48 2 3.3 96.7 
47 3 5.0 93.3 
46 3 5.0 88.3 
45 4 6.7 83.3 
44 3 5.0 76.7 
43 3 5.0 71.7 
42 4 6.7 66.7 
41 4 6.7 60.0 
40 5 8.3 53.3 
39 6 10.0 45.0 
38 3 5.0 35.0 
37 4 6.7 30.0 
36 2 3.3 23.3 
35 3 5.0 20.0 
34 2 3.3 15.0 
33 2 3.3 11.7 
32 2 3:3 8.3 
31 1 1.7 5.0 
30 1 17 3.3 
29 1 1:7 LZ 


“Rounded to the nearest tenth. 


of a frequency polygon in Figure 2.1. Itis customary to use the horizontal axis (also 
called the abscissa, the baseline, or the X-axis) to represent the range of values of the 
variable in question and the vertical axis (called the ordinate or Y-axis) to depict the 
frequencies with which each of the values occut in the distribution. The rules and 
procedures for transforming frequency distributions of various types into graphs 
are presented in most textbooks on basic statistics (see, e.g., Kirk, 1999). 


Numerical Description of Data 


In addition to helping us visualize data through graphs, descriptive statistics also 
provides tools that allow for the properties of data to be summarized numeri- 
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Frequency 


29 31 33 35 37 39 41 43 45 47 49 


Test Scores 
Figure 2.1 Frequency polygon for test scores in Table 2.3 (n = 60) 


cally. These tools describe the central tendency and the variability of numerical 
data. 


Measures of Central Tendency 

One of the first things one wants to know when inspecting a data setis where the 
bulk of the data can be located, as well as the data’s most representative or central 
value. The principal measures of central tendency—the mode, median, and 
mean—tell us these things. As with any other statistics, each of these measures 
has particular advantages and disadvantages depending on the types of data and 
distributions one wishes to describe. Their relative merits and disadvantages, 
which are beyond the scope of this work, are also discussed in most statistics text- 
books (see, e.g., Howell, 2002). 


* The mode, or most frequently occurring value in a distribution, is useful 
primarily when dealing with qualitative or categorical variables. Strictly 
speaking, there can be only one mode or—if there is no variability in 
a distribution—no mode at all. However, if two or more values in a 
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distribution are tied with the same maximum frequency, the distribu- 
tion is said to be bimodal or multimodal. 

© The median (Mdn) is the value that divides a distribution that has been 
arranged in order of magnitude into two halves. If the number of values 
(n) in the distribution is odd, the median is simply the middle value; if 7 
is even, the median is the midpoint between the two middle values. 

e The mean or arithmetic average (U for a population mean, and M for a 
sample mean) is obtained by summing all the values in a distribution and 
dividing the total by the number of 

cases in the distribution. Thus, its 
actual value may or may not be 
In the pages to come you will en- represented in the data set. In spite 


counter a few statistical formulas. If of this, and of the fact that it is the 
you are tempted to skip them, 
DON'T. Remember, this is a book on 
the Essentials of Psychological Testing. 
The only formulas you will encounter mean has many desirable proper- 
in the book are those that convey ties that make it the most widely 


concepts essential for understanding d fread indi f 
testing and the meaning of test scores. used centra tendency MdIcatorTor 


measure of central tendency most 
influenced by extreme scores, the 


quantitative variables. 


Measures of Variability 

These statistics describe how much dispersion, or scatter, there is in a set of data. 
When added to information about central tendency, measures of variability help 
us to place any given value within a distribution and enhance the description of a 
data set. Although there are many measures of variability, the main indexes used 
in psychological testing are the range, the semi-interquartile range, the variance, 
and the standard deviation. 


* The range is the distance between two extreme points—the highest and 
lowest values—in a distribution. Even though the range is easily com- 
puted, it is a very unstable measure as it can change drastically due to 
the presence of one or two extreme scores. 

° The semi-interquartile range is one half of the interquartile range (IQR), 
which, in turn, is the distance between the points that demarcate the 
tops of the first and third quarters of a distribution. The first quartile 
point (Q,), or 25th percentile, marks the top of the lowest quarter 
(quartile) of the distribution. The third quartile point (Q,), ot 75th per- 
centile, is at the top of the third quarter of the distribution and marks 
the beginning of the top quartile. The interquartile range is the range 
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between Q, and Q,; therefore, it encompasses the middle 50% of a dis- 
tribution. In the example presented in Table 2.3, the 25th percentile 

is at the score of 37 and the 75th percentile is at 44. The interquartile 
range is 44 — 37 = 7 and the semi-interquartile range is 7 + 2 = 3.5. 
Note that whereas 53% of the scores fall within a narrow 8-point range, 
the other 47% of the scores are spread over all the remaining range of 
14 score points. 

° The variance is the sum of the squared differences or deviations between 
each value (X) in a distribution and the mean of that distribution (M), 
divided by N. More succinctly, the variance is the average of the sum of 
squares (SS). The sum of squares is an abbreviation for the sum of the 
squared deviation values or deviation scores, X&( X — M)’. Deviation 
scores have to be squared before being added, in order to eliminate neg- 
ative numbers. If these numbers were not squared, the positive and neg- 
ative deviation scores around the mean would cancel each other out 
and their sum would be zero. The sum of squares represents the total 
amount of variability in a score distribution and the variance (SS/N ) 
represents its average variability. Due to the squaring of the deviation 
scores, however, the variance is not in the same units as the original dis- 
tribution. 

° The standard deviation is the square root of the variance. Along with the 
variance, it provides a single value that is representative of the individ- 
ual differences or deviations in a data set-—computed from a common 
reference point, namely, the mean. The standard deviation is a gauge of 
the average variability in a set of scores, expressed in the same units as 
the scores. It is the quintessential measure of variability for testing as 
well as many other purposes and is useful in a variety of statistical ma- 
nipulations. 


Rapid Reference 2.3 lists some of the basic notation symbols that will be used 
in this book, along with the formulas for the mean, interquartile range, standard 
deviation, and variance. The measures of central tendency and variability for the 
60 test scores from Table 2.3 are listed in Table 2.4. Although the detailed infor- 
mation about the 60 scores is lost, the statistics in Table 2.4 concisely describe 
where the scores cluster as well as the average amount of dispersion in the data 
set. 


The Importance of Variability 
Whereas it may be true that variety is the spice of life, it is the very meat of testing, 
The psychological testing enterprise depends on variability across individuals. 
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Basic Notation and Formulas 


X = A single data point or value in a distribution; in psychological testing, X 
almost always stands for a raw score. 


x = Sum of. 


n = Sample size, that is, the total number of cases in a distribution; in psy- 
chological testing, n almost always represents number of people or 
number of scores. 


N = Population size. 


= xX 
M, or X = Mean of X = —— 
n 
u = Population mean 
Mdn = Median = 50th percentile. 
Q, 
Q, = 3rd quartile point = 75th percentile. 
Q, — Q; = Interquartile range (IQR). 
IQR + 2 = Semi-interquartile range. 


Ist quartile point = 25th percentile. 


x(X -MP 
s? = Sample variance = aa 
i 
X(X — )? 
o° = Population variance = we 


s or SD = Sample standard deviation = V3 
© = Population standard deviation = V o? 


Table 2.4 Descriptive Statistics for the 60 Test Scores from Table 2.3 


Measutes of Central Tendency Measutes of Variability 

Mean = 40.13 Range = 50 — 29 =21 
Median = 40.00 Variance = 25.745 
Mode = 39 Standard deviation = 5.074 
Q, or 25th percentile = 37 Interquartile range = Q,-Q, = 44-37= 7 


Q, or 75th percentile = 44 Semi-interquartile range = 7 + 2 = 3.5 


< ESSENTIAL STATISTICS FOR TESTING 51 


Without individual differences there 7 p 
would be no variability and tests Putting It Into Practice 


would be useless in helping us to E A ON 


make determinations or decisions many scores are within + | SD of 
about people. All other things being the mean—that is, between 40 
T5, 


equal, the greater the amount of vari- 


ability there is among individuals, in > M aunne owi thet il ow arine é 


scores, roughly 2/3 of them, are be- 


whatever characteristic we are at- ‘ween OS anal 25. 
tempting to assess, the more accu- e This proportion is typical for distri- 
rately we can make the distinctions butions that approximate the shape 


of the normal curve like the distri- 


that need to be made among them. ies aa 
bution in Figure 2.1. 


Knowing the shapes of score distrib- 
utions as well as their central tenden- 
cies and variabilities provides the basis for a good portion of the essentials of test 
score interpretation discussed in Chapter 3. 


THE NORMAL CURVE MODEL 
Definition 


The normal curve, also known as the bel curve, is a distribution that in some ways 
is similar to the one in Figure 2.1. Its baseline, equivalent to the X-axis of the dis- 
tribution in Figure 2.1, shows the standard deviation (O) units; its vertical axis, or 
ordinate, usually does not need to be shown because the normal curve is nota fre- 
quency distribution of data but a mathematical model of an ideal or theoretical 
distribution. The height to which the curve rises at every single point along the 
baseline is determined by a mathematical formula that describes the specific re- 
lationships within the model and establishes the exact shape and proportions of 
the curve. Like all ideal models, the normal curve does not exist; it is based on 
probability theory. Fortunately, for our purposes, one can understand the basic 
facts about the normal curve without knowing much about its mathematical 
bases. 

Although the normal curve model is an ideal, it is often approximated by the 
distributions of real data, such as the data from Table 2.3, presented in Figure 2.1. 
The similarity between the model and the distributions of many variables in the 
natural world has made it useful in descriptive statistics. Even more important is 
the fact that many chance events, if repeated a sufficiently large number of times, 
generate distributions that approximate the normal curve. It is this connection to 
probability theory that makes the normal curve serve an important role in infer- 
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ential statistics. As we shall see, the usefulness of the normal curve model derives 
from its properties to which we now turn. 


Properties of the Normal Curve Model 


Many of the properties of the normal curve model are clearly evident upon visual 
inspection (see Figure 2.2). For instance, it can be seen that the normal distribu- 
tion has each of the following properties: 


Percent of cases 
under portions of the 
normal curve 


It is bel shaped, as its nickname indicates. 

It is bilaterally symmetrical, which means its two halves are identical (if we 
split the curve into two, each half contains 50% of the area under the 
curve). 

It has tails that approach but never touch the baseline, and thus its Amits 
extend to + infinity (+20), a property that underscores the theoretical and 
mathematical nature of the curve. 

It is unimodal; that is, it has a single point of maximum frequency or 
maximum height. 

It has a mean, median, and mode that coincide at the center of the distribu- 
tion because the point where the curve is in perfect balance, which is 
the mean, is also the point that divides the curve into two equal halves, 
which is the median, and the most frequent value, which is the mode. 


Standard Deviations -40 -30 -20 -lo 0 +16 +20 +30 +40 
| | | | l | | 
Percentile Equivalents 1 5 10 20 | 30 40 50 60 701 80 90 95 99 
a Mdn a3 


Figure 2.2 The normal curve with percentages of cases in each unit o segment 
from —4 to +4, cumulative percentages, and percentile equivalents 
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In addition to these properties, the normal curve has other, less obvious char- 
acteristics that are linked to its mathematical function rule. This formula—which 
is not essential—is available in most statistics textbooks and in some of the nor- 
mal curve Web sites mentioned in Rapid Reference 2.4. It involves two constant 
elements (T and e) and two values that can vary. Each particular normal curve is 
just one instance of a family of normal curve distributions that differ as a func- 
tion of the two values of each specific curve that can vary. The two values that can 
vary are the mean, designated as u, and the standard deviation, designated as ©. 
Once the U and O parameters for a normal distribution are set, one can calculate 
the height of the ordinate (Y-axis), at every single point along the baseline (X- 
axis), with the formula that defines the curve. When the normal curve has a mean 
of zeto and a standard deviation of 1, it is called the standard normal distribution. 
Since the total area under the normal curve equals unity (1.00), knowledge of the 
height of the curve (the Y-ordinate) at any point along the baseline or X-axis al- 
lows us to calculate the proportion (p) or percentage (p X 100) of the area under 
curve that is above and below any X value as well as between any two values of X. 
The statistical table resulting from these calculations, which shows the areas and 
ordinates of the standard normal curve, is available in Appendix C, along with a 
basic explanation of how the table is used. 

In anormal curve, the standard deviation or O units are positioned at equal dis- 
tances along the X-axis, at points that mark the inflections of the curve itself (i.e., 
the points where the curve changes direction). Figure 2.2 shows the normal curve 
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Normal Curve Web Sites 


These are just three of the many Web sites that can be found by entering “the 
normal curve” in a good online search engine: 


* http://www.ms.uky.edu/~mai/java/stat /GaltonMachine.html| 


This site has a simple and visually appealing demonstration of the process that re- 
sults in the bell curve. 


* http://stat-www.berkeley.edu/~stark/Java/NormHiLite.htm 


This Web page has an interactive tool that lets you highlight any segment of the 
normal curve and see immediately what percentage of the area is in the high- 
lighted segment. 


e http://www.psychstat.smsu.edu/introbook/sbk| |.Atm 


This is one of several sites that explains basic facts about the normal curve clearly 
and succinctly. 
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divided at every 6 unit from —4 to +4 as well as the percentages of the area com- 
prised in each segment. Note that if you add all the percentages in the areas above 
the mean, the result equals 50%, as does the sum of all the areas below the mean. 
Furthermore, the area between +16 and —10 is 68.26% (34.13% x 2)—roughly 
2/3 of the curve—and the area between +20 and —20 is 95.44%, almost the en- 
tire curve. Knowledge of these basic facts about the normal curve is exceedingly 
useful in statistics. 


USES OF THE NORMAL CURVE MODEL 
Descriptive Uses 


Since the proportions of the area under the standard normal curve that lie above 
and below any point of the baseline or between any two points of the baseline are 
pteestablished—and easy to find in the tables of areas of the normal curve such 
as the one in Appendix C—we can readily apply these proportions to any other 
distribution that has a similar shape. In testing, this particular application of the 
normal distribution is used repeatedly in generating the standard scores described 
in the next chapter. 

Under some circumstances, even when a distribution approximates but does 
not exactly match the normal curve, we can still use the proportions within the 
normal curve model to normalize scores. Normalizing scores involves transform- 
ing them so that they have the same meaning, in terms of their position, as if they 
came from a normal distribution. This procedure, which is not as complicated as 
it may appear, makes use of the cumulative percentages computed from a fre- 
quency distribution (see Table 2.3). It will be discussed in more detail, with ex- 
amples, in the next chapter. 


Inferential Uses of the Normal Curve Model 


In inferential statistics the normal curve model is useful for (a) estimating popu- 
lation parameters and (b) testing hypotheses about differences. Applications of 
the normal curve model to the estimation of population parameters and to hy- 
pothesis testing make use of two interrelated notions, namely, sampling distribu- 
tions and standard errors. 

Sampling distributions are hypothetical, as opposed to real, distributions of val- 
ues predicated on the assumption that an infinite number of samples of a given 
size could be drawn from a population. If this were done, and if the statistics for 
those samples were recorded, many (but not all) of the resulting distributions of 
statistics or sampling distributions would be normal. The mean of each hypo- 
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Table 2.5 Standard Error of the Mean for the Data from Tables 2.3 
and 2.4 


Mean (M) = 40.13 Standard deviation (s) = 5.074 Sample size (7) = 60 


s 5.074 5.074 
Standard error of the mean (SE,,) V Jeo 7.7459 = 
n 6 ; 


655 


thetical sampling distribution would 


equal the population parameter and DON’T FORGET 


the standard deviation of the sam- 


À oo Appendix C contains the Table of Ar- 
pling distribution would be the stan- eas and Ordinates of the Normal 
dard error of the statistic in question. Curve, along with an explanation of 

The standard error (SE) of an ob- how tt is used. As is true of all the for- 

i pend of mulas in this book, the information in 
tained sample statistic is thus con- Appendix C is presented only be- 
ceived of as the standard deviation cause familiarity with it is an essential 
of the sampling distribution that requirement for understanding test 


; . scores. 
would result if we obtained the same 


statistic from a large number of ran- 

domly drawn samples of equal size. It can easily be calculated using sample sta- 
tistics (see, e.g., Table 2.5). Once we have obtained a given statistic and its stan- 
dard error from a sample, the assumption of a normal sampling distribution 
allows us to use the areas of the normal curve to estimate the population para- 
meter based on the obtained statistic. 


Estimating Population Parameters 


A Hypothetical Example 
In order to estimate a population parameter, such as the average height ofall adult 
women in the United States, we might obtain a randomly drawn sample of 50 
adult women, one from each state. Suppose that the mean height for that sample, 
which would be an estimate of the population mean, is 64 inches and suppose that 
the standard deviation is 4 inches. If we were to repeat this procedure over and 
over, drawing an infinite number of samples of 50 women each and recording the 
means of those samples each time, the sampling distribution of means that would 
result would fit the normal curve model. The mean of that theoretical sampling 
distribution could be construed as the population mean (i.e., the average height 
of all adult women in the United States). 

Obviously, such a course of action is not only impractical, but also impossible. 
Therefore, we use inferential statistics to estimate the population mean. We find 
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the standard error of the mean (SE,,) with the formula s/ Vn, where sis the standard 
deviation of the sample (4) and is the number of cases in the sample (50). In this 
case, dividing 4 by the square root of 50 equals 7.07, which makes the SE = 0.565. 
Thus, based on our obtained sample statistics, we could say that the actual average 
height of adult women in the United States is within the range of our obtained 
mean of 64 in. 0.565 in., or between 63.435 and 64.565 in. Adding and subtract- 
ing 1 SE, from the sample mean gives us a 68% confidence interval for the pop- 
ulation mean, because 68% of the area under the normal curve is within +10 (or 
in this case +1 SE). If we wanted to make a statement with a higher level of con- 
fidence, we could choose a larger confidence interval by selecting a larger number 
of © units and multiplying it times the SE,,. As we see in Figure 2.2, the segment 
between +26 encompasses 95.44% of the area under the normal curve; therefore, 
in our example, the interval between 64 +2 SE), or 64 £2 (0.565 in.) = 64 £1.13 
in. and encompasses the range of 62.87 to 65.13 in. within which we could be 
95.44% confident that the average height of adult women in the United States lies. 

Example with data from Table 2.3: If we calculate the standard error of the mean 
(SE) for the data in Table 2.3, using the formula s/ Vi, where sis the standard 
deviation (5.074) and zis the number of cases in the sample (60), the SE,, = .655 
(see Table 2.5). If the sample of 60 students had been drawn at random from all 
the students who ever took that particular test, we could then assume that there 
is approximately a 68% probability that the mean for the population of a//the stu- 


dents who took that test is within +.655 points, or +1 SH,,, of the obtained mean 


DON’T FORGET 


Two Essential Concepts of Inferential Statistics: 
Sampling Distributions and Standard Errors 


e Sampling distributions are theoretical distributions of the values of a variable, or 
of a statistic, that would result if one were to collect and record values (e.g., 
scores) or statistics (e.g., means, standard deviations, correlation coefficients, 
etc.) for an infinite number of samples of a given size from a particular popula- 
tion. Sampling distributions do not actually exist; they are hypothetical con- 
trivances that are used for assigning probability estimates to obtained values or 
statistics through a device known as the standard error. 


Standard errors are statistical entities that can be computed, by various formu- 
las, on the basis of sample data; they provide the means by which obtained val- 
ues or sample statistics are compared to their theoretical sampling distribu- 
tions. A standard error is the estimated standard deviation of the theoretical 
sampling distribution of an obtained value or statistic. 
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of 40.13, or somewhere in the range of 39.475 and 40.785. Similarly, we can say 
with 95.44% confidence—meaning that our chances of being wrong are less than 
5%—that the interval between the mean of 40.13 +2 SE y, that is, the range from 
38.82 to 41.44, includes the population mean. 


The Significance of Standard Errors 

Standard errors are extremely important in inferential statistics. In both of the ex- 
amples just presented, we could estimate ranges within which population para- 
meters are likely to be found based on the assumptions that (a) the obtained 
sample mean is the best estimate we have of the population mean, and (b) the 
standard error of the mean is the equivalent of the standard deviation of the hy- 
pothetical sampling distribution of means, assumed to be normal. Similar as- 
sumptions, along with the estimates afforded by the areas under the standard nor- 
mal curve and other theoretical distributions that appear in statistics textbook 
tables—such as the Student’s ¢ distribution—can be used not only to generate 
probability statements about population parameters derived from other sample 
statistics, but also to generate probability statements about obtained differences 
between sample statistics themselves. 

When differences between sample means or proportions are tested for signif- 
icance, the obtained differences are divided by the standard errors of those dif- 
ferences, calculated by formulas appropriate to the specific type of difference to 
be tested. The resulting ratios, called critical ratios, along with the appropriate dis- 
tributions for the statistics in question, can then be used to ascertain the proba- 
bility that an obtained difference could have resulted by chance. Although most 
of the techniques of inferential statistics are well beyond the scope of this book, 
we shall encounter standard errors again in connection with the reliability and va- 
lidity of test scores, in Chapters 4 and 5. Rapid Reference 2.5 summarizes the 
main reasons why the normal curve model is so important in the field of psycho- 
logical testing. 


Non-normal Distributions 


Creating graphic representations of obtained distributions permits a comparison 
of the obtained frequency distributions to the normal distribution. This matters 
a great deal because, to the extent that a frequency polygon or histogram differs 
in shape from the normal curve, the proportions of area under the curve no 
longer apply. Furthermore, the particular way in which distributions differ from 
the normal curve may carry significant implications concerning the data. 

There are many ways in which distributions can differ from the normal curve 
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Why Is the Normal Curve So Important in 
Psychological Testing? 


In testing, the normal curve model is used in ways that parallel the distinction be- 
tween descriptive and inferential statistics: 


|. The normal curve model is used descriptively to locate the position of scores 
that come from distributions that are normal. In a process known as normaliza- 
tion, described in Chapter 3, the normal curve is also used to make distribu- 
tions that are not normal—but approximate the normal—conform to the 
model, in terms of the relative positions of scores. 


2. The normal curve model is applied inferentially in the areas of (a) reliability, 
to derive confidence intervals to evaluate obtained scores and differences 
between obtained scores (see Chapter 4), and (b) validity, to derive confi- 
dence intervals for predictions or estimates based on test scores (see 
Chapter 5). 


model. The manner and extent to which they deviate from it has implications 
with regard to the amount of information the distributions convey. An extreme 
case can be illustrated by the distribution that would result if all values in a set of 
data occurred with the same frequency. Such a distribution, which would be rec- 
tangular in shape, would imply no difference in the likelihood of occurrence of 
any given value and thus would not be useful in making decisions on the basis of 
whatever is being measured. 

A different, and more plausible, type of deviation from the normal curve 
model happens when distributions have two ot more modes. If a distribution is 
bimodal, or multimodal, one needs to consider the possibility of sampling prob- 
lems or special features of the sample. For example, a distribution of class grades 
in which the peak frequencies occur in the grades of A and D, with very few B or 
C grades, could mean that the students in the class are atypical in some way or that 
they belong to groups that differ significantly in preparation, motivation, or abil- 
ity level. Naturally, information of this nature would almost invariably have im- 
portant implications; in the case of this example, it might lead a teacher to divide 
the class into sections and use different pedagogical approaches with each. 

Two other ways in which distributions may deviate from the normal curve 
model carry significant implications that are relevant to test data. These devia- 
tions pertain to the properties of the kurtosis and skewness of frequency distrib- 
utions. 
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Kurtosis 

This rather odd term, which stems from the Greek word for convexity, simply 
refers to the flatness or peakedness of a distribution. Kurtosis is directly related 
to the amount of dispersion in a distribution. Platykurtic distributions have the 
greatest amount of dispersion, manifested in tails that are more extended, and /ep- 
tokurtic distributions have the least. The normal distribution is wesokurtic, mean- 
ing that it has an intermediate degree of dispersion. 

Kurtosis Applied: The Hypothesis of Greater Male Variability. In the field of differen- 
tial psychology, a long-standing hypothesis has held that the range ofintelligence 
is greater among males than it is among females. The hypothesis arose from ob- 
servations concerning the overrepresentation of males in the ranks of people of 
extraordinary accomplishments as well as in institutions for the mentally re- 
tarded. Although there has been much debate about this hypothesis and some 
support for it (see, e.g., Halpern, 1997; Hedges & Nowell, 1995), for a variety of 
reasons—including the nature of intelligence tests themselves—the issue is not 
settled. If the hypothesis of greater male variability proved to be accurate, it 
would mean that more males than females are at the extreme high and low ends 
of the distribution of intelligence test scores. In sucha case, the distributions of 
intelligence test scores for females and males would differ in kurtosis. Their 
graphic representations, if superimposed, might look like the hypothetical dis- 
tributions in Figure 2.3, which shows a leptokurtic distribution for females and 


N — Male 


==>- Female 


Frequency 


Low Average High 


Intelligence Test Scores 


Figure 2.3 Hypothetical distributions of intelligence test scores, showing 
greater male than female variability (platykurtic vs. leptokurtic curve) 
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a platykurtic one for males, with no difference in the average scores of males and 
females. 


Skewness 

The skewness (SR) of a distribution refers to a lack of symmetry. As we have seen, 
the normal distribution is perfectly symmetrical, with S& = 0; its bulk is in the 
middle and its two halves are identical. A skewed distribution is asymmetrical. If 
most of the values are at the top end of the scale and the longer tail extends to- 
ward the bottom, the distribution is negatively skewed (Sk < 0); on the other hand, 
if most of the values are at the bottom and the longer tail extends toward the top 
of the scale, the distribution is positively skewed (Sk > 0). 

Skewness Applied. The meaning of skewness with regard to test score distributions 
is easy to see. Ifa distribution is negatively skewed, it means that most people ob- 
tained high scores; if it is positively skewed, it means that most scored in the low 
end. Figure 2.4 displays examples of negatively and positively skewed distribu- 
tions. Panel A of the figure shows a positively skewed distribution of scores on a 
test in which most of the students scored low; Panel B shows a negatively skewed 
distribution of scores on a test in which most test takers scored high. 


Why Is the Shape of Distributions Relevant to Psychological Testing? 

When a test is under development, the shape and characteristics of the score dis- 
tributions obtained with its preliminary versions help to determine what adjust- 
ments, if any, need to be made to the test. The shape of the score distributions that 
are obtained during the process of test development should conform to expecta- 
tions based on what the test is measuring and on the type of test takers who make 
up the preliminary or standardization samples. For example, if an achievement 
test is aimed at college-level students and the distribution of scores from a repre- 
sentative sample of college students is negatively skewed, it means that the test 
may be too easy and the test developer may need to add more difficult items to 
the test in order to bring the bulk of the scores more toward the center of the dis- 
tribution. Conversely, if the same test is given to a representative sample of ele- 
mentary school students and the distribution of their scores is positively skewed, 
the result conforms to expectations and does not suggest a need for readjust- 
ment. 


ESSENTIALS OF CORRELATION AND REGRESSION 


So far, our discussion has dealt primarily with the description and treatment of 
statistics derived from measures of a single variable, or univariate statistics. If we 
were interested only in test scores themselves (an unlikely possibility), those sta- 
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Figure 2.4 Skewed distributions 


tistics would suffice. However, in order for test scores to have some meaning in 
the practical arena, they need to provide information about other variables that 
are significant in the real world. Correlational methods are the techniques used to 
obtain indexes of the extent to which two or more variables are related to each 
other, indexes that are called correlation coefficients. Correlational methods are the 
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major tools we have to demonstrate linkages (a) between scores on different tests, 
(b) between test scores and nontest variables, (c) between scores on parts of tests 
or test items and scores on whole tests, (d) between part scores or item scores and 
nontest variables, and (e) between scores on different parts of a test or different 
items from a single test. Because of these multiple applications, the notion of cor- 
relation plays an important part in the forthcoming discussions of reliability, va- 
lidity, and test development in later chapters. 

With correlation we enter the realm of bivariate or multivariate statistics. Instead 
of having a single frequency distribution of measures on one variable we need at 
least two sets of measurements or observations on the same group of people (e.g., 
scores on two different tests) or matched pairs of observations for two sets of in- 
dividuals (e.g., the scores of twin pairs on the same test). When we compute a cor- 
relation coefficient to describe the relationship between two variables, data are 
organized in the form of bivariate distributions, like the ones presented in Table 
2.6 for two sets of fictitious data. 

In order to compute a correlation coefficient all we need is the data (i.e., ob- 
servations) on two variables. The variables might be annual income and years of 
education for a set of people, the amount of rain and the size crops for a period 
of several years, the average length of women’s skirts and the performance of the 
stock market over a period of time, the position of the sun in the sky and the 
amount of daylight in a given location, the scores ona test and an index of job per- 
formance for a group of employees, or any others. Two variables (usually labeled 
X and Y) can be correlated using any one of several correlational methods that 
differ with regard to the kinds of data and the kinds of relationships for which 
they are appropriate. 


Linear Correlation 


The relationship between two variables is said to be ear when the direction and 
rate of change in one variable are constant with regard to the changes in the other 
variable. When plotted on a graph, the data points for this type of relationship 
form an elliptical pattern that is straight or nearly so. If there is a correlation be- 
tween two variables and the relationship between them is linear, there are only 
two possible outcomes: (a) a positive correlation or (b) a negative correlation. If 
there is no correlation, the data points do not align themselves into any definite 
pattern or trend and we may assume that the two sets of data do not share a com- 
mon soutce of variance. If there is either a positive or a negative correlation of 
any magnitude, we can evaluate the possibility that the correlation could have re- 
sulted from chance, using the size of the sample on which the correlation was 
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Table 2.6 Two Sets of Bivariate Data 


Individual Score on Test X Score on Test Y 


A. Data for a perfect positive correlation 


1 3 5 
2 4 6 
3 6 8 
4 7 9 
5 8 10 
6 9 11 
7 10 12 
8 11 13 
9 13 15 
10 14 16 
B. Data for a perfect negative correlation 
1 140 5 
2 130 6 
3 110 8 
4 100 9 
5 90 10 
6 80 11 
7 70 12 
8 60 13 
9 40 t5 
10 30 16 


computed and statistical tables that show the probability that a coefficient of a 
given magnitude could have occurred by chance. Naturally, the larger the coeffi- 
cient, the less likely it is that it could be the result of chance. If the probability that 
the obtained coefficient resulted from chance is very small, we can be confident 
that the correlation between X and Y is greater than zero. In such cases, we as- 
sume that the two variables share a certain amount of common variance. The 
larger and the more statistically significant a correlation coefficient is, the larger 
the amount of variance we can assume is shared by X and Y. The proportion of 
variance shared by two variables is often estimated by squaring the correlation 
coefficient (r) and obtaining the coefficient of determination, or r? . Although coef- 
ficients of determination tell us how much of the variance in Y can be explained 
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by the variance in X, or vice versa, they do not necessarily indicate that there is a 
causal relationship between X and Y. 


Scatterplots 


The graphic depiction of bivariate data in the form of scatter diagrams or scat- 
terplots is essential in order for us to visualize the kind of relationship at hand. 
The scatterplots in Figure 2.5 present the patterns of points that result from plot- 
ting the bivariate distributions from Table 2.6. These figures let us literally see the 
strength and direction of the relationship between the two variables in each set. 
We can see that in both parts of the figure, the patterns fall in a straight diagonal 
line, indicating that both of the plotted relationships are linear and strong; in fact, 
the correlations are perfect, something that is rarely seen with real data. A strong 
correlation means that as the values of one variable increase or decrease, there is 
a corresponding amount of change in the values of the other variable. The direc- 
tion of the pattern of points in a scatterplot tells us whether the corresponding 
changes are in the same or opposite directions. In the perfect pattern depicted in 
Figure 2.5, Panel A, the relationship is invariant: For every unit increase in the 
scores of Test X, there is a corresponding increase of one unit in the scores of Test 
Y. In Figure 2.5, Panel B, we see another perfect, invariant pattern: For every de- 
crease of 10 units in Test X, there is a corresponding increase of one unit in Test 
Y. The relationships are in opposite directions but both of them maintain a per- 
fect correspondence relative to their respective scales. 


The Discovery of Regression 


The reader may recall from Chapter 1 that Francis Galton made significant con- 
tributions to the development of psychometrics. One of the most important of 
these was Galton’s discovery of the phenomenon he called regression, a discovery 
that resulted from his attempts to chart the resemblance between parents and off- 
spring on a number of variables and to produce evidence of their hereditary na- 
ture. In terms of the variable of height, for instance, Galton discovered that par- 
ents who were taller than average tended to have children who, as adults, were also 
taller than the average height of the parents in his sample, but closer to that aver- 
age than the parents themselves. The reverse was true for parents who were 
shorter than the average; their children, as adults, also tended to be shorter than 
the average height of parents in Galton’s sample, but closer to that average than 
the parents themselves. When he plotted these bivariate data of matched sets 
of heights of patents and children, as well as other sets of variables, Galton 
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A. Perfect positive correlation, r= +1.00 


Scores on Test Y 


Scores on Test X 


B. Perfect negative correlation r=—1.00 


Scores on Test Y 


80 100 120 140 160 


Scores on Test X 


Figure 2.5 Scatterplots of bivariate data from Table 2.6 
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discerned that this pattern of regression toward the mean kept repeating itself: Ex- 
treme scores of parents on one variable tended to be associated with scores that 
were closer to the mean in the offspring, Furthermore, Galton found that if he 
plotted the heights of offspring at various ranges, relative to the average heights 
of parents within those intervals, he obtained a linear pattern he called the regres- 
sion line. Galton understood that the slope of the regression line represented the 
strength or magnitude of the relationship between the heights of parents and chil- 
dren: The greater the slope of the line, the stronger the relationship between the 
two variables. 

In spite of the significance of his discovery, Galton’s conclusions about the 
phenomenon of regression were not quite accurate (see Cowles, 2001). This was 
partly a result of restrictions in the data he used in his analyses and partly due to 
his misinterpretation of the causes of correlations between variables. Given that 
the genetic bases of heredity were unclear at the time when Galton was working 
on these problems, his misinterpretation of regression is understandable. Never- 
theless, the procedures he developed to portray the relationship between vari- 
ables have proved to be extremely useful in assessing the amount of variance 
shared by variables. More importantly, regression analyses have given us a basis 
for making predictions about the value of Variable Y, based on knowledge of the 
corresponding value of Variable X, with which Variable X has a known and sig- 
nificant degree of correlation. After all, it is true that if a set of parents is taller 
than the average, we can also expect their children to be taller than average. Gal- 
ton himself devised a way to quantify the relationship between variables by trans- 
forming the values of each into a common scale and computing a numerical in- 
dex or coefficient that summarized the strength of their relationship. However, 
it was Karl Pearson, a mathematician and disciple of Galton, who refined the 
method and developed the most widely used formula for computing correlation 
coefficients. 


CORRELATION COEFFICIENTS 


As we have seen, correlation simply refers to the extent to which variables are re- 
lated. The degree and the direction of the correlation between variables is mea- 
sured by means of various types of correlation coefficients, which are numbers that 
can fluctuate anywhere from —1.00 to +1.00. Rapid Reference 2.6 lists some other 
basic but often misunderstood facts concerning correlation coefficients in gen- 
eral. 

Unlike the so-called hard sciences, where experimentation is the typical way to 
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Three Essential Facts About Correlation in General 


|. The degree of relationship between two variables is indicated by the number in the 
coefficient, whereas the direction of the relationship is indicated by the sign. 
A correlation coefficient of —0.80, for example, indicates exactly the same de- 
gree of relationship as a coefficient of +0.80. Whether positive or negative, a 
correlation is low to the extent that its coefficient approaches zero. Although 
these facts may seem obvious, the apparently compelling nature of negative 
signs often causes people to forget them. 


2. Correlation, even if high, does not imply causation. 
If two variables, X and, are correlated, it may be because X causes Y, because 
Y causes X, or because a third variable, Z, causes both X and Y. This truism is 
also frequently ignored; moderate to high correlation coefficients are often 
cited as though they were proof of a causal relationship between the corre- 
lated variables. 


3. High correlations allow us to make predictions. 
While correlation does not imply causation, it does imply a certain amount of 
common or shared variance. Knowledge of the extent to which things vary in 
relation to one another is extremely useful. Through regression analyses we 
can use correlational data on two or more variables to derive equations that 
allow us to predict the expected values of a dependent variable (Y), within a 
certain margin of error, based on the known values of one or more indepen- 
dent variables (X,, X,,...%X,), with which the dependent variable is corre- 
lated. 


proceed, in the behavioral sciences the ability to manipulate variables is often re- 
stricted. Thus, research in psychology relies on correlational methods to a great 
extent. Fortunately, the array of research designs and methods of analysis that can 
be applied to data has grown immensely with the power and availability of mod- 
ern computers. Some of the currently commonplace techniques for the simulta- 
neous analysis of data from multiple variables, such as multiple regression and 
path analysis, are so sophisticated that they allow psychologists and other social 
scientists to make some inferences about causal relationships with a high degree 
of confidence. 

Which statistical technique is used to compute a coefficient of correlation de- 
pends on the nature of the variables to be correlated, the types of scales used in 
measuring them, and the pattern of their relationship. Once again, a full review 
of methods is beyond the scope of this book. However, the most widely used in- 
dex of the amount of correlation between two variables bears some discussion. 
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Pearson Product-Moment Correlation Coefficient 


The basic formula that Karl Pearson devised for computing the correlation coef- 
ficient of bivariate data from a sample is formally known as the Pearson product- 
moment correlation coefficient. The definitional formula of this coefficient, more com- 
monly referred to as the Pearson z is 


xx 
= (2.1) 
: NSS, 
where 
T, = the correlation between X and Y; 


= the deviation of a score X from the mean of X scores; 


the deviation of a corresponding Y score from the mean of Y 

scores; 

È xy = the sum of all the cross-products of the deviations (i.e., the sum of 
the products of each x deviation times its corresponding y devia- 
tion); 

= the number of pairs in the bivariate data set; 

s. = the standard deviation of the X scores; and 

the standard deviation of the Y scores 


f 


a 
j 


Although the computational raw-score formula for the Pearson r is more 
complicated than the definitional formula, the easy availability of computer soft- 
ware to compute correlation coefficients makes the computational formula prac- 
tically unnecessary. On the other hand, Formula (2.1) and the even shorter For- 
mula (2.2) are of considerable help in understanding the numerical meaning of 
the correlation coefficient. Rapid Reference 2.7 lists the basic notation for corre- 
lation along with two versions of the formula for the Pearson r. 

The Pearson r is actually the mean of the cross-products of the standard scores 
of the two correlated variables. The formula that embodies this definition is 


222s 2.2 
hy Ty (2.2) 
where 
r= the correlation between X and Y; 


j 


the standard scores of variable X, obtained by dividing each devia- 
tion score on X by the standard deviation of X; and 

the standard scores of variable Y, obtained by dividing each devia- 
tion score on Y by the standard deviation of Y 


N 
j 
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Basic Notation for Correlation 


Variable X 


X = A score on variable X 


x = X-X = Deviation score on X 
S, = Standard deviation of X 
z, = Standard score on variable X 


Variable Y 


Y = A score on variable Y 


y=Y- Y = Deviation score onY 
5 = Standard deviation of Y 
“= Standard score on variable Y 


X-X Yey 
ZL = AES 
4 S. x S; 


Formulas for the Pearson r: 


x xy ff 
= Formula (2.1), definitional formula 
xy 


DE, Ze 
p= N £ Formula (2.2), standard score formula 


where N = number of paired observations of X andY used to compute r. 


Coefficient of determination = r, 


Summing the cross-products of the z scores of the X and Y variables and divid- 
ing by the number of paits in a data set produces an average that reflects the 
amount of relationship between X and Y, namely, the Pearson z: 

Formula (2.2) is of interest in the context of psychological testing, not just be- 
cause of its brevity and conceptual basis, but also because it serves to introduce 
the notion of standard scores, ot z scores, with which we shall deal again in the next 
chapter. The reader may have noticed that the values along the baseline of the 
normal curve in the Table of Areas of the Normal Curve presented in Appendix 
C ate given in terms of z scores. The reason for this is that a z score represents the 
distance between each value in a distribution and the mean of that distribution, 
expressed in terms of the standard deviation unit for that distribution. The stan- 
dard score formula (2.2) for the Pearson r simply provides a more compact way 
of expressing the relationship between two variables. 
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Conditions Necessary for the Use of the Pearson r 

Although it is, by far, the most widely used correlation coefficient, the Pearson r 
is appropriate only for data that meet certain conditions. Since Pearson devel- 
oped his formula, many different methods have been developed to obtain corre- 
lation coefficients for various types of bivariate data. The derivation of the Pear- 
son product-moment correlation coefficient rests on the following assumptions: 


1. The pairs of observations ate independent of one another. 

2. The variables to be correlated are continuous and measured on inter- 
val or ratio scales. 

3. The relationship between the variables is linear; that is, it approximates 
a straight-line pattern, as described earlier. 


Whether the first and second of these assumptions or conditions have been met 
is easily ascertained from knowledge of the manner in which the data were col- 
lected and of the type of data at hand. If the pairs of scores or observations to be 
correlated are obtained independently of one another, the first assumption has 
been satisfied. If the data for both variables represent continuous quantities, the 
second assumption has been met. 

Satisfying the third, and most critical, assumption of the Pearson rrequites in- 
spection of the scatterplot of the bivariate data to see whether the distribution of 
cases falls into the elliptical shape that is indicative of a linear relationship por- 
trayed in Figure 2.6, Panel A. When this assumption is violated, the Pearson ris 
not an accurate index of correlation. 


Deviations from Linearity 
For purposes of determining the applicability of the Pearson r to a set of bivariate 
data, there are two ways in which a scatterplot can deviate from the elliptical shape 
that indicates a linear positive relationship. The first and most obvious way is if there 
is a significant bending of the elliptical shape, as in Figure 2.6, Panels B and C. Such 
deviations indicate that there is no longer a linear relationship and, therefore, that 
the relationship between X and Y is not the same throughout the range of their val- 
ues. The second way in which scatterplots can deviate from the ellipse that indicates 
a linear relationship is a condition called hereroscedasticity. This simply means that the 
dispersion or variability in the scatterplot is not uniform throughout the range of 
values of the two variables. In order to use the Pearson r correlation coefficient, the 
scatterplot needs to show a fairly uniform amount of dispersion, or homoscedasticity, 
throughout the range. The scatterplot in Figure 2.6, Panel A, is homoscedastic, 
whereas the ones in Figure 2.6, Panels D and E, are heteroscedastic. 

One of the ways to avoid the inappropriate application of the Pearson ris by 
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Scatterplots D and E show unequal variabity (heteroscedasticity); D shows greater 
variability at the high end of the range where as E shows greater variability at the low end. 
Figure 2.6 Scatterplots illustrating various characteristics of bivariate data 


Note. Each point marks the location of one pair of observations, or scores on X and Y. 


producing a scatterplot of the bivariate data and inspecting its shape for possible 
deviations from linearity. If the Pearson r formula is applied to data that deviate 
from a straight linear relationship, either in terms of a bending in the shape of the 
scatterplot or due to heteroscedasticity, the resulting correlation coefficient will 
be an incorrect index of the relationship between X and Y. 
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Range Restriction and Correlation 

An important and often overlooked feature of the Pearson r concerns the way it 
is affected by the variability of the correlated variables. Stated simply, the effect 
of a restriction in the range of either one of the variables is to reduce the size of r. 

Example 1: An extreme case. The most extreme, though not very realistic, case of 
range restriction would be a situation in which there is no variability at all in one 
of the correlated variables. If we refer to the definitional formula for the Pearson 
r presented in Rapid Reference 2.7, we can easily see that if there were no vari- 
ability in the scores on either X or Y (i.e., if all the values of either X or Y were the 
same), all the deviation scores of the respective variable and the numerator of the 
Pearson rcoefficient formula would be zero, thereby resulting in a correlation co- 
efficient of zero. This is just one instance of the singular importance of variabil- 
ity highlighted earlier in this chapter. 

Example 2: The effect of range restriction in employment testing. If all the people who 
applied for a large number of available positions in a new corporation were hired, 
regardless of their scores on a preemployment aptitude test, chances are that we 
would find a fairly high correlation between their scores and measutes of job pro- 
ductivity obtained a few months after they were hired. Since we can assume that 
a large group of applicants would exhibit fairly broad ranges both in aptitude test 
scores and in job productivity, the relationship between aptitude and productiv- 
ity would most likely be reflected by the correlation coefficient. If, after a while, 
the personnel selection process gets more restrictive—so that only those appli- 
cants who obtain high scores on the aptitude test are hired—the net effect of this 
change would be to constrict the range of ability among newly hired employees. 
Thus, if a new coefficient were computed only with data for the newly hired, the 
degree of correlation between aptitude test scores and productivity would be re- 
duced. The scatter diagram in Figure 2.7 represents the high positive correlation 
between aptitude test scores and job productivity there might be among the large 
heterogeneous group of people initially hired. The small segment in the upper 
right-hand portion of the diagram represents the low, almost nonexistent, corre- 
lation there would probably be in the much mote restricted group of top appli- 
cants hired later on. 

Just as a restriction in the range of correlated variables will lower the correla- 
tion between them, a wide range of variability in the correlated variables will tend 
to augment the size of the obtained correlation coefficient and possibly overesti- 
mate the relationship between the two variables. The fact that correlation coeffi- 
cients depend on the variability of the samples within which they are found em- 
phasizes the importance of examining the composition of samples from the point 
of view of their appropriateness. Although some statistical corrections for range 
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Figure 2.7 The effect of restricted range on correlation 


restrictions can be used when the variability in a sample is reduced, there is no 
substitute for making sure that the variability of the samples within which coeffi- 
cients are computed corresponds to the variability of the group or groups to 
which the obtained correlations will be applied. 


Other Correlational Methods 

The Pearson r can be used in a wide range of cases, as long as the necessary con- 
ditions are met. When they are not met, other procedures can be applied to ob- 
tain correlations for bivariate data. For example, when the variables to be corre- 
lated are in ordinal form, the correlation method of choice—already mentioned 
in connection with ordinal scales—is Spearman’s rank-difference coefficient of 
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correlation, commonly known as Spearman’s rho (r,). If the relationship between 
two variables is curvilinear, the correlation ratio—commonly known as e/a (‘H)— 
may be used. When one of the variables to be correlated is dichotomous, the point 
biserial correlation, Ot f}, 18 US ed, whereas if both variables are dichotomous, the phi 
ot fourfold coefficient (Q) is employed. Dichotomous variables often come into play 
in analyzing test item data recorded in terms of pass-fail or true-false responding. 

There are many other types of correlation coefficients that ate suited to spe- 
cific types of data. They can be found in statistics textbooks when the need arises. 
One particularly important variant is the mu/tple correlation coefficient (R), which is 
used when a single dependent variable (Y) is correlated with two or more com- 
bined predictors (X,, X,,...,X,). 


CONCLUSION 


This chapter has presented the basic statistical concepts needed to understand 
test scores and their meaning. Statistics exist to help us make sense of data, but 
they do not answer questions. In order to do that we have to use our judgment 
along with the statistics. We will encounter these concepts again in the context of 
the various technical aspects of tests—such as normative information, reliability, 
and validity—that allow us to evaluate their quality as instruments of psycholog- 
ical measurement. 


el TEST YOURSELF AAA 


|. Which of the following scales of measurement is the only one that has a 
meaningful zero point? 


(a) Nominal 
(b) Ratio 

(c) Ordinal 
(d) Interval 


2. Tom and Jerry scored at the 60th and 65th percentiles, respectively, on a 
language skills test. Mary and Martha scored at the 90th and 95th per- 
centiles, respectively, on the same test. We can conclude that the differ- 
ence between Tom and Jerry in terms of their language skills is the same 
as the difference between Mary and Martha. The preceding statement is 


(a) true. 
(b) false. 
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. On atest of general cognitive ability, a 5-year-old child obtains a mental 
age score of 4 years and a 10-year-old child obtains a mental age score of 
9 years. If one were to compute their IQs according to the original ratio 
1Q formula, the result would be as follows: 


a) Both children would obtain the same ratio IQ. 
b) The 5-year-old would obtain a higher ratio IQ. 
c) The 10-year-old would obtain a higher ratio IQ. 


. In the distribution 2, 2, 2, 2,3,3,3,8, I 1, the mean, median, and mode are, 
respectively, 


a) 4, 3, and 2. 
D 24h @iaral 3, 
c) 3,4, and 2. 
d) 2, 3, and 4. 


. For testing and many other purposes, the quintessential index of the vari- 
ability in a distribution of scores is the 


a) sum of the squared deviation scores. 
b) the square root of the variance. 
c) the semi-interquartile range. 


. Which of the following statements about the normal curve model is not 
true? 


a) It is bilaterally symmetrical. 
b 
€ 


ts limits extend to infinity. 


ts mean, median, and mode coincide. 


d) It is multimodal. 


. The area of a normal distribution between +I ø and -I ø encompasses 
approximately of the curve. 


a) 50% 
b) 68% 
c) 95% 
d) 99% 
. If the shape of the distribution of scores obtained from a test is signifi- 


cantly skewed, it means that the test is probably for the test takers 
in question. 


(a) too easy 
(b) too hard 
(c) either too easy or too hard 
(d) just right 


(continued ) 
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9. Which of the following coefficients represents the strongest degree of 
correlation between two variables? 


(a) —80 
(b) —20 
(c) +.20 
(d) +.60 


10. If the range of values of either one of two variables that are correlated 
using the Pearson product-moment coefficient of correlation (Pearson r) 
is restricted, the size of the obtained coefficient will be 


(a) reduced. 
(b) increased. 
(c) unaffected. 


Answers: |.b; 2. b; 3.¢4.a;5.b; 6. d; 7.b; 8.¢9.a; 10.a 


Three 


ESSENTIALS OF TEST 
SCORE INTERPRETATION 


o matter how many statistics are used in psychological testing, in the fi- 

nal analysis the meaning of test scores derives from the frames of refer- 

ence we use to interpret them and from the context in which the scores 
ate obtained. To be sure, test scores also need to be reliable and test items need to 
be carefully developed and evaluated so that they contribute to the meaning of 
scores; these ate matters with which we deal in Chapters 4 and 6. Presently, we 
turn to the frames of reference for interpreting test scores, a topic that is closely 
connected to the validity of inferences we can make on the basis of tests, dis- 
cussed at length in Chapter 5. The context in which testing takes place, an issue 
of paramount importance that is related to the process of test selection and test 
administration, is discussed in the final chapter. Rapid Reference 3.1 lists three ex- 
cellent sources of information where many of the topics discussed in this chapter 
are covered in greater detail. 


RAW SCORES 


A raw score is a number (X) that summarizes or captures some aspect of a person’s 
performance in the carefully selected and observed behavior samples that make 
up psychological tests. By itself, a raw score does not convey any meaning. High 
scores may be a favorable result on tests of ability, but not favorable on tests that 
evaluate some aspect of psychopathology. In the Minnesota Multiphasic Person- 
ality Inventory (MMPI) for instance, elevated scores usually indicate some kind 
of maladjustment yet low scores do not necessarily indicate good adjustment. 
Even if we know the type of test from which a score was obtained, we can be mis- 
led. Some tests of cognitive ability—in particular, many neuropsychological in- 
struments—are scored in terms of number of errors or speed of performance, so 
that the higher the score, the less favorable the result. Moreover, we cannot even 
know how high “high” is without some kind of frame of reference. A score that 
sounds high—such as an IQ of 130, for example—may have quite different 
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For more extensive information on 
technical aspects of many of the topics 
discussed in this chapter, see any one 
of the following sources: 


* Angoff, W. H. (1984). Scales, norms, 
and equivalent scores. Princeton, NJ: 
Educational Testing Service. 


e Petersen, N. S., Kolen, M.J., & 
Hoover, H. D. (1989). Scaling, norm- 
ing, and equating. In R. L. Linn (Ed.), 
Educational measurement (3rd ed., 
pp. 22 1-262). New York: American 
Council on Education/Macmillan. 


e Thissen, D., & Wainer, H. (Eds.). 
(2001). Test scoring. Mahwah, NJ: 
Erlbaum. 


meanings depending on the test from 
which it was derived, the areas the 
test covers, and how recent its norms 
are, as well as specific aspects of the 
situation in which the score was ob- 
tained and the characteristics of the 
test taker. 


FRAMES OF REFERENCE 
FOR TEST-SCORE 
INTERPRETATION 


Underlying all other issues regarding 
score interpretation, in one way or 
another, is the matter of the frames of 
reference used to interpret a given 
score. Depending on their purpose, 


tests rely on one or both of the following sources of information to derive frames 


of reference for their meaning: 


1. 


Norms. Norm-referenced test interpretation uses standards based on the per- 
formance of specific groups of people to provide information for in- 
terpreting scores. This type of test interpretation is useful primarily 
when we need to compare individuals with one another or with a refer- 
ence group in order to evaluate differences between them on whatever 
characteristic the test measures. The term zors refers to the test per- 
formance or typical behavior of one or more reference groups. Norms 
are usually presented in the form of tables with descriptive statistics — 
such as means, standard deviations, and frequency distributions—that 
summarize the performance of the group or groups in question. When 
norms are collected from the test performance of groups of people, 
these reference groups are labeled normative or standardization samples. 
Gathering norms is a central aspect of the process of standardizing a 
norm-referenced test. 

Performance criteria. When the relationship between the items or tasks of 
a test and standards of performance is demonstrable and well defined, 
test scores may be evaluated via criterion-referenced interpretation. This 
type of interpretation makes use of procedures, such as sampling from 
content domains or work-related behaviors, designed to assess 
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whether and to what extent the desired levels of mastery or perfor- 
mance criteria have been met. 


NORM-REFERENCED TEST INTERPRETATION 


Norms are, by far, the most widely used frame of reference for interpreting test 
scores. The performance of defined groups of people is used as a basis for score 
interpretation in both ability and personality testing. When norms are the frame 
of reference, the question they typically answer is “How does the performance of 
this test taker compare to that of others?” The score itself is used to place the test 
taker’s performance within a preexisting distribution of scores or data obtained 
from the performance of a suitable comparison group. 


Developmental Norms 


Ordinal Scales Based on Behavioral Sequences 

Human development is characterized by sequential processes in a number of be- 
havioral realms. A classic example is the sequence that normal motor develop- 
ment follows during infancy. In the first year of life, most babies progress from 
the fetal posture at birth, through sitting and standing, to finally walking alone. 
Whenever a universal sequence of development involves an orderly progression 
from one behavioral stage to another —more advanced—stage, the sequence it- 
self can be converted into an ordinal scale and used normatively. In such cases, the 
frame of reference for test score interpretation is derived from observing and 
noting certain uniformities in the order and timing of behavioral attainments 
across many individuals. The pioneer in the development of this type of scales 
was Arnold Gesell, a psychologist and pediatrician who published the Gesell De- 
velopmental Schedules in 1940 based on a series of longitudinal studies con- 
ducted by him and his associates at Yale over a span of several decades (Ames, 
1989). 

The Provence Birth-to-Three Developmental Profile. A carrent example of an instru- 
ment that uses ordinal scaling is the Provence Birth-to-Three Developmental 
Profile (“Provence Profile”), which is part of the Infant-Toddler Developmental 
Assessment (IDA; Provence, Erikson, Vater, & Palmieri, 1995). The IDA is an in- 
tegrated system designed to help in the early identification of children who are 
developmentally at risk and possibly in need of monitoring or intervention. 
Through naturalistic observation and parental reports, the Provence Profile pro- 
vides information about the timeliness with which a child attains developmental 
milestones in eight domains, in relation to the child’s chronological age. The de- 
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velopmental domains are Gross Motor Behavior, Fine Motor Behavior, Rela- 
tionship to Inanimate Objects, Language/Communication, Self-Help, Relation- 
ship to Persons, Emotions and Feeling States (Affects), and Coping Behavior. For 
each of these domains, the profile groups items into age brackets ranging from 0 
to 42 months. The age brackets are as small as 2 months at earlier ages and as wide 
as 18 months in some domains at later ages. Most span between 3 and 6 months. 
The number of items in each age group differs as well, as does the number of 
items that need to be present or competently performed to meet the criterion for 
each age bracket. Table 3.1 lists four sample items from each of three develop- 
mental domains of the IDA’s Provence Profile. The scores on items at each age 
range and in each domain are added to arrive at a performance age that can then be 
evaluated in comparison to the child’s chronological age. Discrepancies between per- 
formance and chronological age levels, if any, may then be used to determine the 
possible presence and extent of developmental delays in the child. 


Theory-Based Ordinal Scales 
Ordinal scales may be based on factors other than chronological age. Several the- 
ories, such as Jean Piagets proposed stages of cognitive development from in- 


Table 3.1 Sample Items From the Provence Profile of the Infant-Toddler 
Developmental Assessment 


Age Range 
Domain (in months) Item 
Gross Motor Behavior 4to7 Sits alone briefly 
7 to 10 Pulls to stand 


13 to 18 Walks well alone 
30 to 36 Walks up and down stairs 


Language/Communication 4to7 Laughs aloud 
7 to 10 Responds to “no” 
13 to 18 Shows shoe when asked 
30 to 36 Knows rhymes or songs 
Self-Help 4to7 Retrieves lost pacifier or bottle 
7 to 10 Pushes adult hand away 
13 to 18 Partially feeds self with spoon or fingers 
30 to 36 Puts shoes on 


Source: Adapted from the Lnfant-Toddler Developmental Assessment (IDA) Administration Manual by 
Sally Provence, Joanna Erikson, Susan Vater, and Saro Palmeri and reproduced with permission 
of the publisher. Copyright © 1995 by The Riverside Publishing Company. All rights reserved. 
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fancy to adolescence or Lawrence Kohlberg’s theory of moral development, posit 
an orderly and invariant sequence or progression derived at least partly from be- 
havioral observations. Some of these theories have generated ordinal scales de- 
signed to evaluate the level that an individual has attained within the proposed se- 
quence; these tools are used primarily for purposes of research rather than for 
individual assessment. Examples of this type of instrument include standardized 
scales based on Piaget’s delineation of the order in which cognitive competencies 
are acquired during infancy and childhood, such as the Ordinal Scales of Psycho- 
logical Development, also known as the Infant Psychological Development 
Scales (UzZgiris & Hunt, 1975). 


Mental Age Scores 

The notion of mental age scores was discussed in Chapter 2 in connection with 
the ratio IQs of the early Stanford-Binet intelligence scales. The mental age scores 
derived from those scales were computed on the basis of the child’s performance, 
which earned credits in terms of years and months, depending on the number of 
chronologically arranged tests that were passed. In light of the difficulties pre- 
sented by this procedure, described in Chapter 2, this particular way of arriving at 
mental age scores has been abandoned. However, several current tests still pro- 
vide norms that are presented as age equivalent scores and are based on the average 
raw score performance of children of different age groups in the standardization 
sample. 

Age equivalent scores, also known as /est ages, simply represent a way of equat- 
ing the test taker’s performance on a test with the average performance of the 
normative age group with which it corresponds. For example, if a child’s raw 
score equals the mean raw score of 9-year-olds in the normative sample, her or 
his test age equivalent score is 9 years. In spite of this change in the procedures 
used to obtain age equivalent scores, inequalities in the rate of development at dif- 
ferent ages remain a problem when this kind of age norm is used, because the dif- 
ferences in behavioral attainments that can be expected with each passing year di- 
minish greatly from infancy and early childhood to adolescence and adulthood. 
If this is not understood, or if the meaning of a fest age is extended to realms other 
than the specific behavior sampled by the test—as it is, for example, when an 
adolescent who gets a test age score of 8 years is described as having “the mind 
of an 8-year-old”—the use of such scores can be quite misleading. 


Grade Equivalent Scores 

The sequential progression and relative uniformity of school curricula, especially 
in the elementary grades, provide additional bases for interpreting scores in terms 
of developmental norms. Thus, performance on achievement tests within school 
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settings is often described by grade levels. These grade equivalent scores are derived 
by locating the performance of test takers within the norms of the students at 
each grade level—and fractions of grade levels—in the standardization sample. 
If we say, for instance, that a child has scored at the seventh grade in reading and 
the fifth grade in arithmetic, it means that her or his performance on the reading 
test matches the average performance of the seventh-graders in the standardiza- 
tion sample and that, on the arithmetic test, her or his performance equals that of 
fifth-graders. 

In spite of their appeal, grade equivalent scores also can be misleading for a 
number of reasons. To begin with, the content of curricula and quality of in- 
struction vary across schools, school districts, states, and so forth; therefore, 
grade equivalent scores do not provide a uniform standard. In addition, the ad- 
vance expected in the early elementary school grades, in terms of academic 
achievement, is much greater than it is in middle school or high school; thus, just 
as with mental age units, a difference of one year in retardation or acceleration is 
far more meaningful in the early grades than it is by the last years of high school. 
Moreover, if a child who is in the fourth grade scores at the seventh grade in arith- 
metic, it does not mean the child has mastered seventh-grade arithmetic; rather, 
it means that the child’s score is significantly above the average for fourth-graders 
in arithmetic. Furthermore, grade equivalent scores are sometimes erroneously 
viewed as standards of performance that all children in a given grade must meet, 
whereas they simply represent average levels of performance that—due to the in- 
evitable variability across individuals—some students will meet, others will not, 
and still others will exceed. 


DON’T FORGET 


All developmental norms are relative, except as they reflect a behavioral se- 
quence or progression that is universal in humans. 


e Theory-based ordinal scales are more or less useful depending on whether the 
theories on which they are based are sound and applicable to a given segment 
of a population or to the population as a whole. 


e Mental age norms or age equivalent score scales reflect nothing more than the 
average performance of certain groups of test takers of specific age levels, at a 
given time and place, on a specific test. They are subject to change over time, as 
well as across cultures and subcultures. 


* Grade-based norms or age equivalent score scales also reflect the average perfor- 
mance of certain groups of students in specific grades, at a given time and 
place. They too are subject to variation over time, as well as across curricula in 
different schools, school districts, and nations. 
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Within-Group Norms 

Most standardized tests use some type of within-group norms. These norms essen- 
tially provide a way of evaluating a person’s performance in comparison to the per- 
formance of one or more appropriate reference groups. For proper interpretation 
of norm-referenced test scores it is necessary to understand the numerical proce- 
dures whereby raw scores are transformed into the large variety of derived scores that 
are used to express within-group norms. Nevertheless, it is good to keep in mind 
that all of the various types of scores reviewed in this section serve the simple pur- 
pose of placing a test taker’s performance within a normative distribution. There- 
fore, the single most important question with regard to this frame of reference 
concerns the exact make-up of the group or groups from which the norms are de- 
rived. The composition of the normative or standardization sample is of utmost 
importance in this kind of test score interpretation because the people in that 
sample set the standard against which all other test takers are measured. 


The Normative Sample 

In light of the important role played by the normative sample’s performance, the 
foremost requirement of such samples is that they should be representative of the 
kinds of individuals for whom the tests are intended. For example, if a test is to 
be used to assess the reading skills of elementary school students in Grades 3 to 
5 from across the whole nation, the normative sample for the test should repre- 
sent the national population of third-, fourth-, and fifth-graders in all pertinent 
respects. The demographic make-up of the nation’s population on variables like 
gender, ethnicity, language, socioeconomic status, urban or rural residency, geo- 
graphic distribution, and public- or private-school enrollment must be reflected 
in the normative sample for such a test. In addition, the sample needs to be suf- 
ficiently large as to ensure the stability of the values obtained from their perfor- 
mance. 

The sizes of normative samples vary tremendously depending on the type of 
test that is standardized and on the ease with which samples can be gathered. For 
example, group ability tests used in school settings may have normative samples 
numbering in the tens or even hundreds of thousands, whereas individual intelli- 
gence tests, administered to a single person at a time by a highly trained examiner, 
are normed on much smaller samples—typically consisting of 1,000 to 3,000 in- 
dividuals —gathered from the general population. Tests that require specialized 
samples, such as members of a certain occupational group, may have even smaller 
normative samples. The recency of the normative information is also important 
if test takers are to be compared with contemporary standards, as is usually the 
case. 


84 ESSENTIALS OF PSYCHOLOGICAL TESTING +) 


CAUTION 
Although the three terms are often used interchangeably—here and else- 
where—and may actually refer to the same group, strictly speaking, the precise 
meanings of standardization sample, normative sample, and reference group are 
somewhat different: 


e The standardization sample is the group of individuals on whom the test is origi- 
nally standardized in terms of administration and scoring procedures, as well as 
in developing the test's norms. Data for this group are usually presented in the 
manual that accompanies a test upon publication. 


e The normative sample is often used as synonymous with the standardization 
sample, but can refer to any group from which norms are gathered. Additional 
norms collected on a test after it is published, for use with a distinct subgroup, 
may appear in the periodical literature or in technical manuals published at a 
later date. See, for example, the study of older Americans by lvnik and his asso- 
ciates (1992) at the Mayo Clinic wherein data were collected to provide norms 
for people beyond the highest age group in the standardization sample of the 
Wechsler Adult Intelligence Scale-Revised (WAIS-R). 


e Reference group, in contrast, is a term that is used more loosely to identify any 
group of people against which test scores are compared. It may be applied to 
the standardization group, to a subsequently developed normative sample, to a 
group tested for the purpose of developing local norms, or to any other desig- 
nated group, such as the students in a single class or the participants in a re- 
search study. 


Relevant factors to consider in the make-up of the normative sample vary de- 
pending on the purpose of the test as well as the population on which it will be 
used. In the case of a test designed to detect cognitive impairment in older adults, 
for instance, variables like health status, independent versus institutional living 
situation, and medication intake would be pertinent, in addition to the demo- 
graphic variables of gender, age, ethnicity, and such. Rapid Reference 3.2 lists 
some of the most common questions that test users should ask concerning the 
normative sample when they are in the process of evaluating the suitability of a 
test for their purposes. 

Reference groups can be defined on a continuum of breadth or specificity de- 
pending upon the kinds of comparisons that test users need to make to evaluate 
test scores. At one extreme, the reference group might be the general population 
of an entire nation or even a multinational population. At the other end, reference 
groups may be drawn from populations that are narrowly defined in terms of sta- 
tus of settings. 

Subgroup norms. When large samples are gathered to represent broadly defined 
populations, norms can be reported in the aggregate or can be separated into sub- 
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Information Needed to Evaluate the Applicability of 
a Normative Sample 


In order to evaluate the suitability of a norm-referenced test for a specific pur- 
pose, test users need to have as much information as possible regarding the nor- 
mative sample, including answers to questions such as these: 


e How large is the normative sample? 
e When was the sample gathered? 
e Where was the sample gathered? 


e How were individuals identified and selected for the sample? 
e Who tested the sample? 
e How did the examiner or examiners qualify to do the testing? 


e What was the composition of the normative sample, in terms of 
—age! 
—sex? 
—ethnicity, race, or linguistic background? 
—education? 
—socioeconomic status? 
—geographic distribution? 
—any other pertinent variables, such as physical and mental health status or 
membership in an atypical group, that may influence test performance? 


Test users can evaluate the suitability of a norm-referenced test for their specific 
purposes only when answers to these questions are provided in the test manual 
or related documents. 


group norms. Provided that they are of sufficient size—and fairly representative of 
their categories—subgroups can be formed in terms of age, sex, occupation, eth- 
nicity, educational level, or any other variable that may have a significant impact 
on test scores or yield comparisons of interest. Subgroup norms may also be col- 
lected after a test has been standardized and published to supplement and expand 
the applicability of the test. For instance, before the MMPI was revised to create 
the MMPI-2 and a separate form for adolescents (the MMPI-A), users of the 
original test—which had been normed exclusively on adults—developed special 
subgroup norms for adolescents at various age levels (see, e.g., Archer, 1987). 

Local norms. On the other hand, there are some situations in which test users 
may wish to evaluate scores on the basis of reference groups drawn from a spe- 
cific geographic or institutional setting. In such cases, test users may choose to 
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develop a set of /ocal norms, for members of a more narrowly defined population 
such as the employees of a particular company or the students of a certain uni- 
versity. Local norms can be used for evaluating the performance of students or 
employees within a given setting, or for making decisions about school or job ap- 
plicants in relation to the standards of a certain place or institution. 

Convenience norms. Occasionally, for reasons of expediency or financial con- 
straints, test developers use norms based on a group of people who simply hap- 
pen to be available at the time the test is being constructed. These convenience norms 
are of limited use because they are not representative of any defined population; 
they are often composed of individuals who are easily accessible to the test de- 
velopers, such as the students in a college class or the residents of a particular re- 
tirement home. In cases like these, the nature of the normative sample should be 
made clear to potential users of the test. 


Scores Used for Expressing Within-Group Norms 


Percentiles 

Percentile rank scores, already discussed in Chapter 2, are the most direct and 
ubiquitous method used to convey norm-referenced test results. Their chief ad- 
vantages are that they are readily understood by test takers and applicable to most 
sotts of tests and test populations. A percentile score indicates the relative position 
of an individual test taker compared to a reference group, such as the standard- 
ization sample; specifically, it represents the percentage of persons in the refer- 
ence group who scored at or below a given raw score. Thus, higher percentile 
scores indicate higher raw scores in whatever the test measures; the 50th per- 


CAUTION 
Due to the similarity of the two terms, percentile scores are often confused with 
percentage scores. These two types of scores are, in fact, quite different and use 
entirely different frames of reference: 


e Percentiles are scores that reflect the rank or position of an individual's perfor- 
mance on a test in comparison to a reference group; their frame of reference is 
other people. 


e Percentage scores reflect the number of correct responses that an individual ob- 
tains out of the total possible number of correct responses on a test; their 
frame of reference is the content of the entire test. 


One way to avoid confusion is to make it a practice to use the percent symbol 
(%) strictly for percentage scores and use a different abbreviation, such as PR or 
%iile, to designate percentile scores. 
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centile (P) or median, corresponds to the raw score point that separates the top 
and bottom halves of the score distribution of the reference group. In a normal 
distribution the 50th percentile is also the group’s mean level of performance. 

An additional advantage of percentile rank scores comes into play when there 
is more than one normative group for the same test or when normative groups are 
subdivided by categories, such as gender, age, or ethnicity. When additional norms 
ate available, a raw score can be located within the distributions of two or more 
different groups or subgroups and easily converted into percentile ranks. For 
example, interest inventory scores for various occupational groups are often re- 
ported for men and womenas separate sex group norms, so that test takers can see 
their rankings on a given interest and occupational scale compared to both groups. 
This information is particularly useful for those who are considering an occupa- 
tion that is significantly segregated along sex lines, such as engineering or nursing, 
The separation of norms allows individuals to gauge the relative strengths of their 
expressed interests in comparison to members of both sex groups. 

If test scores were evenly distributed throughout their range, resulting in a rec- 
tangular frequency polygon, percentiles would probably be the scores of choice 
in nearly all situations. However, as seen in Figure 2.2 and in Figure 3.1 later in 
this chapter, in a normal distribution the majority of scores tend to cluster around 
a central value and scatter more widely at the extremes. This fact, which is also 
true of many non-normal distributions, means that percentile score units ate usu- 
ally markedly unequal at different points of the range. In a normal or nearly nor- 
mal distribution, such as those obtained from most tests, the percentage of the 
people who score near the middle is much greater than at the extremes. There- 
fore, any given difference in percentile rank score units magnifies the apparent 
discrepancy in the relative performance of individuals whose scores ate in the 
middle range and compresses the apparent extent of the difference in the relative 
performance of individuals at the high and low ends of the distributions. 

Another disadvantage of percentile rank scores pertains to the most extreme 
scores in a distribution. To be sure, for any given normative group there is always 
a score that is the highest and one that is the lowest. As long as scores are inter- 
preted strictly in reference to a specific normative sample, the highest score can 
be said to be at the 100th percentile because all the cases are at or below that score; 
technically, we could even describe the score below the lowest one obtained by 
everyone in a specific sample as the zero percentile score, although this is not 
done ordinarily. In terms of the larger population that the normative sample rep- 
resents, however, the interpretation of such scores is problematic. 

Test ceiling and test floor. The issue of how to accommodate individuals at the 
highest and lowest ends of the spectrum of ability for which a test is designed is 
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Putting It Into Practice 


Using Percentile Rank Scores: 
The Disadvantage of Unequal Units 


The Vocabulary subtest of the Wechsler Adult Intelligence Scale—Third Edition 
(WAIS-III), a test for individuals aged |6 to 89 years, consists of 33 vocabulary 
words. Each word definition may accrue a score of 0, |, or 2 points. Thus, the sub- 
test raw scores can range between 0 and 66, depending on the quality of the re- 
sponses. 


The WAIS-III manual (Wechsler, 1997) displays the performance of the standard- 
ization samples in tables for various age ranges. For individuals between the ages 
of 45 and 54, the table shows that raw scores ranging from 45 to 48 points are 
ranked at the 50th percentile and raw scores from 49 to 5! points are at the 
63rd percentile. In contrast, raw scores between |5 and 19 points rank at the 2nd 
percentile and those between 20 and 24 points rank at the 5th percentile. 


This clearly highlights the problem of inequality of percentile score units: For people in 
the 45- to 54-year-old age group, a difference of only 6 points (45 to 51) in the 
middle of the raw score distribution results in a difference of |3 percentile rank 
units (50th to 63rd percentiles), whereas a raw score difference of 9 points (15 to 
24) at the low end of the range results in a difference of only 3 percentile rank 
units (2nd to 5th percentiles). 


most relevant in the context of test development, discussed in Chapter 6. Never- 
theless, at this point, it is worth noting that the individuals employed in standard- 
izing a test do set the upper and lower limits of performance on that test. Ifa test 
taker reaches the highest score attainable on an already standardized test, it means 
that the /est ceiling, or maximum difficulty level of the test, is insufficient: one can- 
not know how much higher the test taker might have scored if there were addi- 
tional items or items of greater difficulty in the test. Similarly, if a person fails all 
the items presented in a test or scores lower than any of the people in the norma- 
tive sample, the problem is one of insufficient Zesz floor. In cases like these, the in- 
dividuals in question have not been adequately tested. 


Standard Scores 

One way to surmount the problem of the inequality of percentile units and still 
convey the meaning of test scores relative to a normative or reference group is to 
transform raw scores into scales that express the position of scores, relative to the 
mean, in standard deviation units. This can be accomplished by means of simple 
linear transformations. A near transformation changes the units in which scores are 
expressed while leaving the interrelationships among them unaltered. In other 
words, the shape of a linearly derived scale score distribution for a given group of 
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test takers is the same as that of the : a : 
original raw score distribution. A CAUTION 


great advantage of this procedure is As wes eet tie previous chepier 


that the normally distributed scores the statistical formulas and procedures 
of tests with different means, stan- presented here are the essential ones 


dard deviations, and score ranges can neecec ior a ba ounce handing er 
? score transformations. Although the 


be meaningfully compared—and statistical operations described in this 
averaged—once they have been lin- book can be and are routinely carried 
out with computer software programs, 
such as SPSS and SAS, the formulas 
and steps involved must be under- 
group is used. stood in order to achieve a basic grasp 


The first linear transformation of the meaning of various scores. 


performed on raw scores is to con- 


early transformed into a common 
scale, as long as the same reference 


vert them into standard-score deviates, or Z scores. A z score (see Appendix C) ex- 
presses the distance between a raw score and the mean of the reference group in 
terms of the standard deviation of the reference group. It will be recalled that the 
mean and standard deviation of z scores ate zero and 1, respectively, and that the 
distribution of z scores is bilaterally symmetrical, with one half of the cases on 
each side of the mean. The position of z scores relative to the mean is indicated 
by the use of a positive sign (or no sign) for the z scores that are above the mean 
and a negative sign for those below it. Thus, the sign of a z score indicates the di- 
rection in which a score deviates from the mean of a group, whereas its numeri- 
cal value reflects the score’s distance from the mean in standard-deviation units. 
For example, a z score of +1.25 indicates that the original raw score is 14 SD units 
above the mean of the group, whereas a raw score that falls “4 SD units below the 
mean converts into a z score of —0.75. If the distribution of scores for the refer- 
ence sample is normal, z scores can be readily transformed into percentiles by re- 
ferring to the Table of Areas of the Normal Curve presented and explained in 
Appendix C. 

Rapid Reference 3.3 shows the linear transformation formulas used for deriv- 
ing z scores from raw scores and for transforming z scores into the other types of 
standard scores. Because the transformation of raw scores into z scores is usually 
the first one in the process of score transformations, z scores ate considered to 
be the most basic type of standard score and are often identified simply as stan- 
dard scores. This also distinguishes them from other familiar types of derived or 
standard scores, such as IQs, that have become associated with specific tests and 
to which we turn next. 

Additional systems for deriving standard scores. Although z scores allow us to know 
immediately the magnitude and direction of the difference between any given 
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where X = Raw score 
X = Reference group mean 
SD, = Standard deviation (SD) of the reference group 


Formula for transforming z scores into other derived standard 
scores: 


New standard score = (z score) (New SD) + New mean 


Example: To transform a z score of +1.00 to an IQ score with M = 100 and 
SD = 15, 
IQ score = (+1.00) (15) + 100 = 115 


score and the mean of its distribution, they involve negative values and decimals. 
Because of this, z scores usually undergo additional linear transformations. The 
goal of the standard score systems that result from these subsequent transforma- 
tions is simply to express test results in more convenient form. 

The numbers chosen as the means and standard deviations in transforming z 
scores into various other standard score formats are arbitrary. However, through 
their frequent use in the contexts in which they are employed, these score formats 
have become familiar and have acquired certain commonly understood mean- 
ings—which may not always be warranted—for those who use them. Figure 3.1 
displays the normal curve with the baselines for percentiles, z scores, and the fol- 
lowing widely used standard score systems: 


© T-scores (M = 50, SD = 10), used in many personality inventories, such as 
the Minnesota Multiphasic Personality Inventory (MMPD and the Cali- 
fornia Psychological Inventory (CPI). 

* College Entrance Examination Board (CEEB) scores (M = 500, SD = 100), 
used by the College Board’s SAT as well as by the Educational Testing 
Service for many of their graduate and professional school admission 
testing programs, such as the Graduate Record Exam (GRE). 

© Wechsler scale subtest scores (M = 10, SD = 3), used for all the subtests of 


Putting It Into Practice 


Transforming Raw Scores Into g Scores 


|. Assume that all the students in a large eighth-grade class took achievement 
tests in social science, grammar, and math. The scores of the class on all three 
tests were normally distributed, but the tests were different in the following ways: 


Total Number of Items Mean SD 
Social science test 50 35) 
Grammar test 20 |'5 3 
Math test 100 70 10 


2. Further assume that you had a reason to want to compare the scores of three 
of the students in the class in relation to each other and in relation to the en- 
tire class. The three students in question—Alfred, Beth, and Carmen—scored 


as follows: 
Raw Scores 
Alfred Beth Carmen 
Social science test 49 38 48 
Grammar test 15 [2 18 
Math test 68 95 75 


3. These scores cannot be compared or averaged across tests, because they are 
on different scales. To compare the scores, even on a single test, we must refer 
to the means, standard deviations (SDs), and numbers of items of each test. An 
easier way to compare them is to convert each of the raw scores into z 
scores—by subtracting the respective test mean from each raw score and di- 
viding by the corresponding standard deviations, as shown in the formula in 
Rapid Reference 3.3—with the following results: 


z Scores 
Alfred Beth Carmen 
Social science test +2.80 +0.60 +2.60 
Grammar test 0.00 —|.00 +1.00 
Math test —0.20 +2.50 +0.50 
Average grade +0.87 +0.70 srl. 
4. The linear transformations of raw scores into z scores lets us average the three 
grades for each student and compare the students’ performance on all the 


tests to each other and to their whole class. Furthermore, since we assumed nor- 
mal distributions for all the tests, we can use the table in Appendix C to translate 
each z score into a percentile rank score. 
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Percent of cases under 
portions of the normal curve 0.13% 2.14% 
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Figure 3.1 The normal curve, percentiles, and selected standard scores 


Note: Adapted from Test Service Notebook #148 of The Psychological Corporation. 


the Wechsler scales, as well as for the subtests of several other instru- 
ments. 

© Wechsler scale deviation IOs (M = 100, SD = 15), used for the summary 
scores of all the Wechsler scales and other tests, including many that do 
not label their scores as “IQs.” 

* Otis-Lennon School Ability Indices (M = 100, SD = 16), used in the Otis- 
Lennon School Ability Test (OLSAT), which is the current title of the 
seties of group tests that started with the Otis Group Intelligence Scale. 


OLSAT indexes are included in Figure 3.1, as an example of a standard score 
system with a mean of 100 and standard deviation other than 15, to illustrate the 
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arbitrariness of the choice of units in standard score systems. Although most 
standard score systems that use a mean of 100 select 15 as their standard devia- 
tion, there are some that use other standard deviations, such as 12 or 18. These al- 
ternative choices can make a significant difference in the interpretation of scores. 
For example, if two normally distributed tests—both with means of 100—have 
standard deviations of 12 and 15, respectively, a standard score of 112 on the first 
test will transform into a z score of +1.00, and rank at the 84th percentile, whereas 
on the second test a standard score of 112 will transform into a z score of +0.80 
and rank at only the 79th percentile. 

A note about deviation IQs. The scores known as deviation IQs were introduced by 
David Wechsler in 1939 for use in his first intelligence scale, the Wechsler- 
Bellevue I, which later became the Wechsler Adult Intelligence Scale (WAIS). 
These scores came into wider use after the first edition of the Wechsler Intelli- 
gence Scale for Children (WISC) was published in 1949. They are called deviation 
IQs to differentiate them from the original rao IQs used in the Stanford-Binet 
and other scales. Deviation IQs are obtained by adding the scale scores the test 
taker obtains on various subtests and locating this sum in the appropriate nor- 
mative table, rather than by the MA/CA X 100 formula. 

The Wechsler 1Q—type score scale has been adopted by numerous other test 
developers to express the summary scores of various tests, including the Stan- 
ford-Binet’s most recent edition, which uses 15 as its standard deviation unit 
rather than its original standard deviation of 16. Among the multitude of tests 
that employ standard scores with a mean of 100 and standard deviation of 15 are 
all of the major instruments designed to assess general cognitive functioning, 
such as the Kaufman series of tests (e.g., Kaufman-Assessment Battery for Chil- 
dren, Kaufman Adolescent and Adult Intelligence Test, etc.), the Differential 
Ability Scales, the Das-Naglieri Cognitive Assessment System, and many others. 
Although they share the same type of units as the Wechsler tests for their global 
ot summary scores, all of these newer tests have discarded the use of the term JO 
to designate their scores. This is a sensible move because the so-called deviation 
IQs are vot quotients. Furthermore, as can be seen from the titles of some of the 
more recently developed instruments, test authors are increasingly abandoning 
the use of the word żnzelligence to designate the construct assessed by their tests in 
favor of othet, mote neutral, terms. 


Nonlinear Transformations 

Not all score transformations are linear. Nonlinear transformations are those that 
convert a raw score distribution into a distribution that has a different shape 
than the original. This can be done through methods that afford test developers 
greater flexibility in dealing with raw score distributions than linear conversions 


Putting It Into Practice 


Transforming x Scores Into Different Standard Scores 


|. To illustrate the conversion of z scores into different types of standard score 
systems, we return to the previous example of the three students from an 
eighth-grade class whose z scores on three normally distributed achievement 
tests were as follows: 


z Scores 
Alfred Beth Carmen 
Social science test +2.80 +0.60 +2.60 
Grammar test 0.00 —|.00 +1|,00 
Math test —0.20 250 +0.50 


2. To transform these z scores into a more convenient scale, we apply the for- 
mula for converting them into other standard scores, presented earlier in 
Rapid Reference 3.3: 


ew standard score = (z score) (New SD) + New mean 


3. Using the appropriate means and standard deviations for each system, we ob- 
tain the following scores: 


Alfred Beth Carmen 


T scores (M = 50, SD = 10) 


Social science test 78 56 76 

Grammar test 50 40 60 

Math test 48 75 55 
CEEB-type scores (M = 500, SD = 100) 

Social science test 780 560 760 

Grammar test 500 400 600 

Math test 480 750 550 
Wechsler scale IQs (M = 100, SD = 15) 

Social science test 42 109 139 

Grammar test 00 85 WS 

Math test Dy 138 108 


4. Note that because these were all linear transformations, the scores stand in 
exactly the same positions relative to one another regardless of the differences in 
units. This can be seen most clearly in the relationship between T scores and 
CEEB scores, wherein the CEEB mean and standard deviation equal the T 
score mean and standard deviation times |0; therefore, in our example, each 
CEEB score equals the corresponding T score times |0. 
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do. Although some nonlinear score transformations involve complex opera- 
tions that are well beyond our scope in this book, others do not. For instance, 
the transformation of normally distributed raw scores into percentile rank 
scores, which we have already considered, is a nonlinear conversion. It is ac- 
complished by transforming each raw score into a z score and locating the z 
score in the Table of Areas of the Normal Curve to derive the proportion or per- 
centage of the area of the normal curve that is below that point. In this section, 
we discuss two additional and widely used types of nonlinearly derived standard 
scores. 

Normalized standard scores are used when a score distribution approximates but 
does not quite match the normal distribution. To normalize scores one first finds 
the percentage of persons in the reference sample who fall at or below each raw 
score (see, e.g., the Cumulative Percent (CP) column for the distribution of 60 
test scores in Table 2.3). Next, the percentages are converted into proportions. 
These proportions ate then located in the Table of Areas of the Normal Curve to 
obtain the z scores corresponding to those areas (see Appendix C). Standard 
scores derived in this fashion are indistinguishable in form from those obtained by 
the linear transformation formula, but should always be identified as normalized 
standard scores to alert the test user to the fact that they come from a distribution 
that was not normal. Once normalized standard scores are obtained, they can be 
transformed into any of the other convenient standard score systems we have dis- 
cussed, such as T'scores, deviation IQs, or CEEB scores, using the same procedure 
employed with linearly derived z scores that is described in Rapid Reference 3.3. 

Stanines were originally devised by the U.S. Air Force during World War II. The 
“standard nine,” ot stanine, scale transforms all the scores in a distribution into 
single-digit numbers from 1 to 9. This device has the distinct advantage of re- 
ducing the time and effort needed to enter scores on a computer for storage and 
further processing, Stanine transformations also make use of cumulative fre- 
quency and cumulative percentage distributions, such as the ones in Table 2.3; 
stanine scores are allocated on the basis of the percentage of cases at given score 
ranges. Table 3.2, Part A, shows the normal curve percentages within each stanine 
unit and the cumulative percentages used in converting raw scores to stanines. 
Part B contains a few examples of stanine transformations, using some of the 
scores from Table 2.3. Figure 3.1 displays the position of stanine scores within the 
normal curve. As can be seen in the figure, stanine 5 comprises the middle 20% 
of the scores. The mean of the stanine scale is 5 and its standard deviation is ap- 
proximately 2. Even though the stanine scale is economical and simple, its brevity 
and simplicity also result in a certain loss of precision. 
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Putting It Into Practice 


Normalized Standard Scores 


To demonstrate the process of normalizing the scores of a distribution that does 
not conform to the normal curve but approximates it, we can use five of the raw 
scores from the distribution of 60 test scores presented in Table 2.3 in Chapter 2. 


The raw scores selected arbitrarily for this exercise are 49, 40, 39, 36, and 29. 
Steps involved in this conversion are as follows: 


Raw score — Cumulative percent (CP) —> Cumulative proportion (cp) 
— Normalized z score 


|. The raw score and cumulative percent are located in the distribution. 
2. The cumulative proportion is the cumulative percent divided by 100. 


3. A normalized z score is obtained from the Table of Areas of the Normal Curve 
in Appendix C by finding the proportion of the area of the curve that comes 
closest to the cumulative proportion for a given score. For scores with cumula- 
tive proportions above 0.50, the areas in the larger portion—from column (3) 
of the table—must be used to obtain the normalized z scores, which will bear 
a positive sign. For scores with cumulative proportions below 0.50, the areas in 
the smaller portion—from column (4) of the table—must be used, and the 
resulting normalized z scores will have negative signs. 


4. These procedures yield the following results: 


Raw Score Cumulative % cp Normalized z Score? 
49 98.3 0.983 2,2 
40 533 0555 +0.08 
39 45.0 0.450 —0.13 
36 23.3 0.233 —0.73 
a) 1.7 0.017 —),|2 


@Normalized standard scores have the same meaning as linearly derived standard scores in 
terms of the position of the original raw scores they represent, relative to their distributions. 
They can be transformed into various other standard score systems, but must always be iden- 
tified as having been derived nonlinearly from a normalization procedure. 
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Table 3.2 Converting Raw Scores to Stanine Scores 


A. Normal Curve Percentages for Use in Stanine Conversion 


Stanine 
1 2 3 4 5 6 7 8 9 
Percentage of cases 
within each stanine 4 7 12 17 20 17 12 7 4 
Cumulative % at each 
stanine 4 11 23 40 60 77 89 96 100 


B. Selected Test Scores from Table 2.3 Converted to Stanines 


Selected Scores* Cumulative % Stanine Scores 
49 98.3 9 
47 93.3 8 
45 83.3 ef 
43 71.7 6 
41 60.0 5 
39 45.0 5 
37 30.0 4 
35 20.0 3 
33 11.7 3 
31 5.0 2 
29 1.7 1 


*See Table 2.3 in Chapter 2 for the complete distribution of 60 test scores. 


INTERTEST COMPARISONS 


In most circumstances, norm-referenced test scores cannot be compared unless 
they are obtained from the same test, using the same normative distribution. An 
additional reason for lack of comparability of test scores stems from differences 
in scale units, like the various sizes of SD units discussed earlier in connection 
with deviation IQs. Furthermore, even when the tests, the norms, and the scale 
units employed are the same, test scores do not necessarily have the same mean- 
ing. When test scores are used in the context of individual assessment, it must be 
kept in mind that many other factors extraneous to the test may also enter into 
test results (e.g., the test taker’s background and motivation, the influence of the 
examiners, and the circumstances under which the tests were taken). 
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DON’T FORGET 


|. Test scores cannot be meaningfully 
compared if 


e the tests or test versions are dif- 
ferent, 


e the reference groups are differ- 
ent, or 


e the score scales differ, 


except when the tests, groups, or 
scales have been purposefully 
equated. Angoff’s (1984) mono- 
graph on Scales, norms, and equiva- 
lent scores is one of the best 
sources of information on equating 
procedures. 


2. Even when test scores have been 
made comparable, the context in 
which testing takes place, as well as 
the backgrounds of test takers, 
need to be taken into account in 
test score interpretation. 


Equating Procedures 


Notwithstanding the cautions out- 
lined in the preceding section, there 
are a number of situations in which it 
is necessary or desirable to compare 
scores of individuals or groups across 
time, or in various psychological 
functions, against a uniform norm. 
For such situations, test developers 
and publishers have devised several 
ways of achieving some comparability 
of scores across tests. Essentially, 
these are all designed in order to place 
test scores in a common frame of 
reference. An additional benefit of 
equating techniques is that they may 
reduce the considerable cost and time 
involved in standardizing tests. Most 
equating procedures involve highly 
technical details, described in An- 


goff’s (1984) monograph and other sources (e.g., Petersen et al., 1989). The fol- 
lowing brief descriptions of some of the more frequently used approaches may 


provide the reader with basic explanations of what they entail. 


e Alternate forms consist of two or more versions of a test meant to be 
used interchangeably, intended for the same purpose, and administered in 
identical fashion. Creating alternate forms that are alike in the content they 
cover but vary in the specific items they contain is one of the simplest ways 
of producing comparable tests. A stricter form of comparability can be 
achieved with the type of alternate versions of tests known as parallel forms. 
These forms are equated not only in content coverage and procedures, but 
also in some of their statistical characteristics such as raw score means and 
standard deviations, as well as indexes of their reliability and validity. Alter- 
nate forms are especially useful when a person has to take the same test on 
more than one occasion. Practice effects (1.e., score increases attributable to 
prior exposure to test items or to items similar to those in a test) do enter 
into play when an alternate form of a previously taken test is given, but they 
are not quite as great as when the same version of the test is taken more than 
once. 
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e Anchor tests consist of common sets of items administered to different 
groups of examinees in the context of two or mote tests and provide a dif- 
ferent solution to the problem of test score compatability. Having the re- 
sponses of more than one normative group to common sets of items, 
within a given time frame, allows for the use of score equating proce- 
dures—based on statistics derived from the common items—that permit 
extrapolating and comparing the scores on one test to those of another, for 
both individuals and groups. This technique may be used when test devel- 
opets wish to provide for comparisons of levels of performance in differ- 
ent skill areas—such as reading comprehension and written expression— 
from two different tests, on a uniform scale. In recent years, however, this 
sort of comparability is more likely to be achieved either through simulta- 
neous norming or item response theory techniques, to be discussed shortly. 

© Fixed reference groups provide a way of achieving some comparability 
and continuity of test scores over time. This method makes use of anchor 
tests embedded in each successive form of a test to provide a linkage to one 
or mote earlier forms of the same test. In this fashion a test series becomes 
linked through a chain of common items to the scores of the group selected 
as a fixed reference for the purpose of maintaining the continuity of the 
score scale over time. The College Board’s SAT is the best known example 
of a test that has made use of fixed reference groups. Until April of 1995, 
all SAT scores were expressed in terms of the mean and standard deviation 
of the 11,000 college applicants who took the test in 1941; on that scale, a 
score of 500 corresponded to the mean of the 1941 fixed reference group, 
ascote of 400 fell 1 SD below that mean, and so on. After April of 1995, re- 
ported SAT scores reflect a recentering of the score scale on contemporary 
college applicants, so that a score of 500 represents a current average level 
of performance on the test. The use of a fixed reference group on the SAT 
over several decades allowed for the evaluation of increases or declines in 
the caliber of performance of college applicants of different eras, as can be 
seen in Rapid Reference 3.4. In addition to the recentered standard scores, 
the percentile rank scores of college applicants on the SAT still can be, and 
still are, reported using the most recent group of college-bound seniors as 
a reference group. 

© Simultaneous norming of two or more tests on the same standardization 
sample, often referred to as co-norming, is yet another method used to 
achieve comparability of scores. By norming tests at the same time and on 
the same group of people, one can readily compare the performance of in- 
dividuals or subgroups on more than one test, using the same standard. 
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Changing Reference Group Standards by Recentering the SAT 


From the 1940s to the 1990s, the scores of college applicants on both the Verbal 
and Math portions of the SAT had shown significant declines. Thus, after recenter- 
ing took place in the 1990s, the recentered Verbal score of 500 turned out to be 
the equivalent of a score of 420 when compared to the 1941! reference group. 
This shift represents a decline of almost | SD unit. The recentered Math score of 
500 was found to be the equivalent of a score of 470 for the 1941 reference 
group. 

Various reasons have been adduced for these changes. Among those reasons, the 
most plausible ones center on the increased socioeconomic and ethnic diversity 
of college applicants and on changes in the quality of secondary school curricula 
from the early 1940s to the early 1990s. 


This capability is particularly helpful when one wishes to contrast relative 
levels of performance on two or more psychological functions, such as ex- 
pressive and receptive vocabulary levels or short- and long-term memory, 
for the same individual or subgroup. The Woodcock-Johnson II (WJ MI) 
provides an outstanding example of co-norming of two test batteries. The 
WJ II Tests of Cognitive Abilities (WJ II COG) is a battery designed to 
measure both general and specific cognitive functions, whereas the WJ III 
Tests of Achievement (WJ III ACH) battery is meant to assess a person’s 
academic strengths and weaknesses. These two batteries were normed on 
the same large sample of individuals, ranging from preschoolers to older 
adults, representative of the population of the United States, and thus pro- 
vide ample opportunity for comparing intraindividual levels of perfor- 
mance in a number of indexes of cognitive functioning and academic skills. 


Item Response Theory (IRT) 


A variety of sophisticated procedures based on mathematical models are increas- 
ingly replacing the traditional equating techniques that have just been described. 
These procedures, which date back to the 1960s and are also known as /atent trait 
models, ate most often grouped under the label of ¿zem response theory (IRT). The 
term /atent trait reflects the fact that these models seek to estimate the levels of var- 
ious—unobservable—abilities, traits, or psychological constructs that underlie 
the observable behavior of individuals, as demonstrated by their responses to test 
items. Unlike the previously discussed techniques for equating tests and test 
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scores, IRT methods apply mathematical models to test ¿że data from large and 
diverse samples, hence the name ¿zem response theory. These data are used to cali- 
brate test items along one or mote parameters and derive probability estimates of 
the amount of ability, or trait level, needed to respond to each item in a certain 
way. Essentially, these models place both persons and test items on a common 
scale (Embretson & Reise, 2000). 

Additional elements and basic procedures of IRT will be discussed at greater 
length in Chapter 6. In the present context, however, the point worth noting 
about IRT models is that if they meet certain conditions they can produce item 
parameter estimates that are ‘variant across populations. This means that these 
estimates are not necessarily tied to the performance of any specific reference 
group. Instead, item response data can be interpreted in terms of an ability or trait 
dimension. Thus, when IRT models are applied to the arrays of item response and 
test score data from various samples and the assumptions of the models are met, 
they can be used in two reciprocal ways: (a) to estimate the probability that per- 
sons with specified levels of the ability or trait in question will answer an item cor- 
rectly or respond to it in a certain way, and (b) to estimate the trait levels needed 
in order to have a specified probability of responding to an item in a certain way. 


Computerized Adaptive Testing 

One of the main advantages of IRT methodology is that it is ideally suited for use 
in computerized adaptive testing (CAT). When individuals take computerized 
adaptive tests (CATs), their ability levels can be estimated based on their re- 
sponses to test items during the testing process; these estimates are used to select 
subsequent sets of test items that are appropriate to the test takers’ ability levels. 
Although many CATs have a fixed number of items, others ate designed so that 
testing can stop whenever a specified stopping rule is satisfied and trait or ability 
levels have been established with sufficient precision. In any case, CAT proce- 
dures shorten test length and testing time significantly; they can also reduce the 
frustration that many paper-and-pencil test takers may experience when they are 
exposed to test booklets that contain items considerably above, or below, their 
levels of ability. Large-scale testing programs, such as those of the Educational 
Testing Service and the U.S. Department of Defense, have been trying out and 
using CAT methodology for several years (see, e.g., Campbell & Knapp, 2001; 
Drasgow & Olson-Buchanan, 1999). Although this methodology has clear ad- 
vantages over papert-and-pencil tests of fixed length—and its range of applica- 
tions is bound to expand—it does present some new problems related to test 
security, test costs, and the inability of examinees to review and amend their 
responses (Wainer, 2000). 
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Test Revisions 


Test names can easily present a possible source of confusion for potential test 
users who are not sufficiently informed. Taking test titles at face value is rarely 
justified. Some of the discrepancies between the titles of tests and what the tests 
actually assess are fairly obvious: personality inventories do not actually inventory 
personality. Others are not as readily apparent: Aptitude tests may or may not test 
aptitudes, and so forth. 

Even if tests share the same name, they may not be equivalent. Many tests un- 
dergo revisions from time to time and retain the same title, except for the addi- 
tion of a number or letter that identifies a specific version (e.g., WISC, WISC-R, 
WISC-III, WISC-IV). In general, the more successful and widely used tests are 
more likely to be revised than other tests. The objectives and extent of test revi- 
sions may range from small changes in the wording of items to major reorgani- 
zations in content, scoring, or administrative procedures. 

A test that has been modified in any way cannot be considered comparable to 
a previous version, unless the similarity is established empirically. When a test re- 
vision is minor and not expected to affect scores, the comparability of the old and 
revised versions can be established relatively easily. This is usually accomplished 
by giving both versions of the test to the same group of people. If the two ver- 
sions are highly correlated and have similar means, standard deviations, and score 
distributions, for most practical purposes it is assumed that both versions are in- 
terchangeable. A typical instance of this practice is when paper-and-pencil tests, 
like the MMPI-2, are transferred to a computer-based administration format 
without any changes in test content or scoring, 

Major test revisions of norm-referenced tests, on the other hand, require re- 
standardization of the test with a new normative sample. Thus, when the changes 
are significant enough to justify differences in the test’s scale or scoring, one is ac- 
tually dealing with a new and different test, albeit a test that may bear some re- 
semblance to and share the same title as previous versions. A preeminent example 
is the Stanford-Binet Intelligence Scale (S-B), which was first published in 1916. 
The fourth and fifth editions of the S-B, published in 1986 and 2003, respectively, 
are altogether different from its earlier versions in almost every respect and over 
time they have become more similar in form and content to the Wechsler scales 
than to the original S-B. 


Longitudinal Changes in Test Norms 
When a test is revised and standardized on a new sample after a period of sev- 
eral years, even if revisions in its content are minor, score norms tend to drift in 
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one direction or another due to changes in the population at different time pe- 
riods. One such change, discussed in Rapid Reference 3.4, is the decline in aver- 
age SAT scores from the fixed reference group tested in 1941 to the college ap- 
plicants of the 1990s. A puzzling longitudinal trend in the opposite direction, 
known as the Fynn effect, has been well documented in successive revisions of 
major intelligence tests (like the S-B and the Wechsler scales) that invariably in- 
volve the administration of both the old and new versions to a segment of the 
newer standardization sample, for comparative purposes. Data from revisions 
of various intelligence tests in the United States as well as in other countries— 
extensively analyzed by J. R. Flynn (1984, 1987)—show a pronounced, long- 
term upward trend in the level of performance required to obtain any given IQ 
score. The Flynn effect presumably reflects population gains over time in the 
kinds of cognitive performance that intelligence tests sample. A variety of fac- 
tors—such as improved nutrition, better prenatal care, and greater environ- 
mental complexity—have been proposed as reasons for this finding. Still, the 
extent to which the Flynn effect generalizes to populations throughout the 
world—as well as its possible causes where it does appear—remain a subject of 
considerable controversy (Neisser, 1998). 

A pertinent example of longitudinal changes in personality test performance 
was seen in the renorming of the MMPI (originally published in the 1940s), which 
took place in the 1980s as part of the development of the MMPI-2 (Butcher, 
Dahlstrom, Graham, Tellegen, & Kaemmer, 1989). Alterations in the content of 
the inventory were relatively minor. Nevertheless, the shift from the original 
MMPI norms to those of the MMPI-2 resulted in modifications of the score lev- 
els considered clinically significant on various scales due to substantial changes in 
the ways people from the two different time periods responded to the items. 

The kinds of changes just described are commonplace when normative 
samples are updated. They underscore the need for such updating in order to 
maintain the currency of within-group norms. Such changes also explain the rea- 
son why the documentation that accompanies any type of norm-referenced test 
should provide a complete and accurate description of the composition of the 
samples used to standardize and norm it, including the dates when the samples 
were gathered. Test documents should incorporate answets to all the questions 
listed in Rapid Reference 3.2, including a cleat exposition of the steps taken in 
the development of the test and of the administration and scoring procedures 
used during its standardization. Test users, in turn, need to be aware of this in- 
formation and take it into account when selecting tests and interpreting test 
scores. 
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Putting It Into Practice 


How the “Flynn Effect” May Affect Apparent Changes in 
Intelligence Test Scores: An Example Using the Wechsler 
Intelligence Scale for Children (WISC) 


John obtained a Wechsler IQ score of | |7 on the WISC-R at the age of 9 anda 
Wechsler IQ of 107 on the WISC-III at age 13. This seemingly significant decline in 
his performance cannot be taken at face value. 


One factor that may partly explain the apparent decline is the Flynn effect. This 
effect refers to the higher level of performance typically seen in the normative 
groups of newer versions of general intelligence tests compared to their older 
counterparts—for example, the WISC-III, published in 1991, compared to the 
WISC-R, published in 1974. According to the WISC-III manual, a sample of 206 
children aged 6 to |6 were given the WISC-R and WISC-III in counterbalanced 
order, with intervals ranging from |2 to 70 days between the two tests. Their 
average Full Scale IQs were 108.2 on the WISC-R and 102.9 on the WISC-III 
(Wechsler, 1991). 


Although the correlation between the two versions of the WISC for this sample 
was high (.89), the children’s scores on the WISC-III were on average lower—by 
slightly more than 5 points—than those on the WISC-R. The difference in the 
mean IQ scores of this group indicates that the norms of the two tests are such 
that, on the whole, any given IQ score represents a higher level of performance 
on the newer test than on the older one. 


Naturally, many factors besides the Flynn effect could have contributed to the ob- 
tained difference between John's two IQs. Chief among them are the measure- 
ment error in each score (see Chapter 4) and the possibility that John might have 
undergone an actual decline in general intellectual functioning due to some illness 
or life event. 


CRITERION-REFERENCED TEST INTERPRETATION 


In the realm of educational and occupational assessment, tests are often used to 
help ascertain whether a person has reached a certain level of competence in a 
field of knowledge or skill in performing a task. For such cases, the frame of ref- 
erence for test score interpretation must change. Instead of comparing a person’s 
performance to that of others, the performance of an individual, or a group, is 
compared to a predetermined criterion or standard. When used in this context, 
criterion may tefer either to knowledge of a specific content domain or to compe- 
tence in some kind of endeavor. Standards by which criterion-referenced tests are 
evaluated are typically defined in terms of specified levels of knowledge or ex- 
pertise necessary to pass a course, obtain a degree, or get a professional license; 
they may also involve a demonstration of sufficient competence to do a job or to 
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create a product. Often, but not always, the application of criterion-referenced 
tests involves the use of cutoff scores, or score ranges, that separate competence 
from incompetence or demarcate different levels of performance. In such cases, 
the validity of the inferences made on the basis of scores needs to be established 
through empirical links between test scores and performance on the criterion. 


Varieties of Criterion-Referenced Test Interpretation 


The term criterion-referenced testing, popularized by Glaser (1963), is sometimes 
used as synonymous with domain-referenced, content-referenced, objective-referenced, ot 
competency testing. This rather confusing state of affairs is due, in part, to the fact 
that criterion-referenced test interpretation makes use of at least two underlying 
sets of standards: (a) those that are based on the amount of knowledge of a content 
domain as demonstrated in standardized objective tests, and (b) those that are 
based on the /eve/ of competence in a skill area as displayed in the quality of the per- 
formance itself or of the product that results from exercising the skill. Occasion- 
ally, the term criterion-referenced testing is also used to refer to interpretations based 
on the preestablished relationship between the scores on a test and expected lev- 
els of performance on a criterion, such as a future endeavor or even another test. 
In this particular usage, the “criterion” is a specific outcome and may or may not 
be related to the tasks sampled by the test. The latter is in sharp contrast to 
content-referenced or performance-based tests, in which the test tasks are essen- 
tially samples of behavior directly related to the criterion. Rapid Reference 3.5 
lists some of the major points on which norm-referenced testing differs from cri- 
terion-referenced testing, as well as some of the differences among types of 
criterion-referenced testing, 

In addition to the foregoing distinctions, the particular manner in which crite- 
tion-referenced tests are used also varies. Sometimes the criteria are strictly quan- 
titative, as when certain percentages (e.g., 80% or 90%) of correct responses 
needed to establish adequate mastery are set. At other times, the criteria are more 
qualitative and subjective in nature. Furthermore, sometimes performance on 
these tests is evaluated on an all-or-none basis with regard to whether a certain 
level of competence has been achieved, and sometimes there may be allowances 
for intermediate levels of competence. 

In spite of the differences in emphases and nomenclature among criterion-ref- 
erenced tests, these instruments do share some common characteristics. Typi- 
cally, criterion-referenced tests (a) are meant to assess the extent to which test tak- 
ers ate proficient in certain skills or knowledge domains, and (b) are scored in 
such a way that one petson’s performance does not influence the relative stand- 
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Norm- Versus Criterion-Referenced Test Interpretation 


e Norm-referenced tests seek to locate the performance of one or more individu- 
als, with regard to the construct the tests assess, on a continuum created by 
the performance of a reference group. 


* Criterion-referenced tests seek to evaluate the performance of individuals in rela- 
tion to standards related to the construct itself. 


e Whereas in norm-referenced test interpretation the frame of reference is al- 
ways people, in criterion-referenced test interpretation the frame of reference 
may be 


—knowledge of a content domain as demonstrated in standardized, objective 
tests; or 


—level of competence displayed in the quality of a performance or of a 
product. 


e The term criterion-referenced testing is sometimes also applied to describe test 
interpretations that use the relationship between the scores and expected lev- 
els of performance or standing on a criterion as a frame of reference. 


ing of others. Whereas norm-referenced tests seek to rank or place one or morte 
individuals in relation to others with regard to the construct they assess, criterion- 
referenced tests seek to evaluate the performance of individuals in relation to the 
actual construct itself. 

In the present context, only those aspects of criterion-referenced test inter- 
pretation necessary for a basic understanding of its premises and terminology will 
be discussed. Several of these concepts are revisited in greater detail in Chapter 5 
in connection to the topic of validity, with which they are inextricably related. 
Rapid Reference 3.6 lists a few sources of information from the extensive litera- 
ture that is available on various kinds of criterion-referenced testing. 


Testing Knowledge of Content Domains 


In order to use knowledge of content domains as a frame of reference for test 
score interpretation, there needs to be a carefully defined and clearly demarcated 
field or subject from which to derive samples (i.e., test items or tasks) to gauge the 
test taker’s knowledge. Moreover, the objectives to be assessed both in terms of 
knowledge of a content domain and application of that knowledge, as well as the 
standards to be used in assessing those objectives, should stem from a consensus 
of people who are experts in the field. Such a situation is encountered primarily 
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Selected Readings on Criterion-Referenced Testing 


The literature on criterion-referenced testing, dating back to the 1960s, is abun- 
dant. To pursue the topic at greater length, readers may want to consult one or 
more of the following sources: 


e Cizek’s (2001) book on Setting Performance Standards, which focuses on both 
theoretical and practical aspects of standard setting and its many ramifications; 


e Popham and Husek’s (1969) classic article on the implications of criterion- 
referenced measurement; 


e Hambleton and Rogers’ (1991) book chapter on “Advances in Criterion- 
Referenced Measurement,’ which provides a useful introduction to the field 
and a description of technical developments from the 1960s to the 1980s; 


e Wigdor and Green's (1991) edited volume on various aspects of performance 
assessment in the workplace; and 


e Linn's (1994) brief but enlightening clarification of some of the confusion in var- 
ious interpretations of the meaning of criterion-referenced testing. 


Full references on these works are listed in the reference section at the end of 
this book. 


in educational or training settings, where subjects and disciplines tend to be par- 
titioned into lessons, courses, programs of study, and other curricular units to 
which students are exposed and from which content areas and learning outcomes 
can be sampled. These tests are usually described as measures of “achievement.” 
They tend to have items—such as multiple-choice questions—that call for test 
takers to select a response or to complete a highly structured task (such as writ- 
ing a short paragraph on a topic or solving a mathematical problem). 

When knowledge domains are the frame of reference for test interpretation, 
the question to be answered is “How much of the specified domain has the test 
taker mastered?” and scores are most often presented in the form of percentages 
of correct answers. This sort of criterion-referenced test interpretation is often 
described as content- or domain-referenced testing. In fact, some consider these two 
terms as synonymous with criterion-referenced testing. Planning for such tests 
requires the development of a table of specifications with cells that specify the num- 
ber of items or tasks to be included in the test for each of the learning objectives 
and content areas the test is designed to evaluate. The proportion of test items al- 
located to each cell reflects the weight or importance assigned to each objective 
and area. Rapid Reference 3.7 shows examples of various objectives and items in 
two content areas typical of domain-referenced tests. Examples of tables of spec- 
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Examples of Domain-Referenced Test Objectives and Items 


|. Domain: Arithmetic 
A. Content area to be assessed: Multiplication of fractions 
B. Objectives to be assessed: 
|. Knowledge of the steps involved in multiplying fractions 
2. Understanding of the basic principles involved in multiplying fractions 
3. Application of principles in solving fraction multiplication problems 
C. Sample test items for each objective: 
Item |. List the steps involved in multiplying fractions. 
Item 2. Draw a diagram to show 1/4 of |/2 of a pie. 
ltem 3. How much is 3/4 x 1/2? 
Il. Domain: Vocabulary 


A. Content area to be assessed: Word knowledge 
B. Objectives to be assessed: 

|. Word definition 

2. Comprehension of word meaning 

3. Application of word knowledge in written expression 
C. Sample test items for each objective: 

Item |. What does “mariner” mean? 

Item 2. Which word is closest in meaning to ‘mariner’? 


marinate 
marimba 
sailor 
pirate 
e. wanderer 


ano” 


Item 3. Make up a sentence using “mariner” in a meaningful context. 


ifications for content-referenced tests and guidance in how they are prepared can 


be found in Gronlund (2003) and Linn and Gronlund (1995, pp. 119-125). 


Performance Assessment 


For purposes of decision-making in the workplace, and in the realm of education 
as well, there is often a need to ascertain or certify competence in the perfor- 
mance of tasks that are more realistic, more complex, more time consuming, or 
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more difficult to evaluate than those typical of content- or domain-referenced 
testing. This kind of assessment calls for evaluating performance through work 
samples, work products, or some other behavioral display of competence and 
skill in situations that simulate real-life settings. 

When the purpose of an assessment is to ascertain levels of competence in the 
kinds of contexts in which skills are applied in real life, the criterion in “criterion- 
referenced” test interpretation is the quality either of the performance itself or of 
the product that results from applying a skill. In this frame of reference, the typ- 
ical questions to be answered are: “Does this test taker display mastery of the skill 
in question?” or “How proficient is this test taker, or group of test takers, in the 
continuum of competence relevant to this particular endeavor?” 


Evaluation and Scoring in the Assessment of Performance 

In light of the questions that the evaluation of proficiency is designed to answer, 
the assessment of performance entails a different set of procedures than those 
used when testing for knowledge in content domains. In general, the assessment 
of performance tends to rely more heavily on subjective judgment. An exception 
to this rule occurs when criteria can be quantified in terms of speed of perfor- 
mance, number of errors, units produced, or some other objective standard. A 
classic and simple example of an objective type of performance assessment is the 
typing test given to those who apply for clerical jobs that require a good deal of 
typing. Even in this type of test, however, the actual numerical criterion that is 
used as a cutoff score for acceptable performance—for example, 65 words per 
minute with no more than 5 errors—is likely to be set arbitrarily. Most other 
kinds of performance assessments involve (a) identifying and describing qualita- 
tive criteria for evaluating a performance or product, and (b) developing a 
method for applying the criteria. The usual methods for evaluating qualitative cri- 
teria involve rating scales or scoring rubrics (i.e., scoring guides) that describe and il- 
lustrate the rules and principles to be applied in scoring the quality of a perfor- 
mance or product. A well known example of this type of procedure is the scoring 
of athletic performances by designated expert judges in events such as figure skat- 
ing on ice or diving competitions. 

Mastery testing. Procedures that evaluate test performance on the basis of 
whether the individual test taker does or does not demonstrate a preestablished 
level of mastery are known as mastery tests. Many of these tests yield all-or-none 
scores, such as pass or fail, based on some criterion level that separates mastery 
from nonmastery. A typical example with which most readers will be familiar is 
provided by the driving tests many states require for the issuance of a driver’s li- 
cense. In these tests, what matters is whether individuals can demonsttate that 
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they know the rules of driving and ate able to handle an automobile in various 
traffic situations. Moreover, it is expected that the vast majority of people who 
take the driving test will be able to pass it, even if it takes more than one trial, and 
no distinctions need to be made among test takers in terms of their levels of per- 
formance, other than whether they pass it or fail it. 

There are some situations and skills—such as landing an airplane on an air- 
craft carrier or performing brain surgery—when anything short of ascertaining 
that complete mastery has been achieved may not be an option. On the other 
hand, in educational circles the notion of testing for mastery can be interpreted 
in different ways. Some educators and other concerned parties have argued that 
all able students should achieve complete mastery of the prescribed instructional 
objectives at one level before graduating or moving on to the next, regardless of 
how long it takes. However, most people are willing to concede that when testing 
is designed to evaluate mastery of the basic skills that are taught in schools—such 
as reading, writing, and arithmetic—there clearly is room for various levels of 
achievement between mastery and nonmastery. In such cases, the problem be- 
comes one of designating appropriate targets that students are expected to 
achieve, within a continuum of performance, in order to be promoted or gradu- 
ate. 


Predicting Performance 


Sometimes the term criterion-referenced test interpretation is used to describe the appli- 
cation of empirical data concerning the link between test scores and levels of per- 
formance, to a criterion such as job performance or success in a program of study. 
In this context, the term criterion is used in a different sense, and more in accord 
with traditional psychometric practice, than in the preceding examples. Here, the 
criterion is an outcome to be estimated or predicted by means of a test. This type 
of information constitutes the basis for establishing the predictive validity of tests 
to be discussed mote fully in Chapter 5. Nevertheless, it is mentioned at this point 
because, when the relationship between test scores and criteria is used for selec- 
tion or placement of individuals in educational, training, or employment settings, 
that relationship can also be construed as a frame of reference for score interpre- 
tation. In this framework, the questions to be answered with the help of the test 
scores are “What level of criterion performance can one expect from a person who 
obtains this score?” or “Is the test taker’s performance on the test sufficient to as- 
sure the desired level of criterion performance in a given endeavor?” 
Information on the relationship between test scores and criteria can be pre- 
sented in a variety of ways, including correlation coefficients and regression equa- 
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tions, which are described mote extensively in the context of validity in Chapter 
5. For the present purposes, however, two procedures that are especially relevant 
to the sort of criterion-referenced test interpretation discussed in the preceding 
paragraph—namely, expectancy tables and expectancy charts—will serve to 
clarify this approach. 

Expectancy tables show the distribution of test scores for one or more groups of 
individuals, cross-tabulated against their criterion performance. Assuming that 
there is a substantial degree of correlation between test scores and criterion mea- 
sures, this information can be used to gauge the probable criterion standing of in- 
dividuals who scored at different levels. For example, using information from 
previous classes, a teacher might cross-tabulate the test scores at midterm as the 
predictor and final grade on the course as the criterion, as shown in Table 3.3. The 
resulting table can inform future students about what their final grades are likely 
to be based on their test scores at midterm. 

Expectancy charts are used when criterion performance in a job, training pro- 
gram, ot program of study can be classified as either successful or unsuccessful. 
These charts present the distribution of scores for a group of individuals along 
with the percentage of people at each score interval who succeeded (or failed) in 
terms of the criterion. When the trend is such that the percentage of successful 
individuals is much greater among high scorers than among low scorers, charts of 
this type can be extremely useful in making selection decisions. 


Relationship Among the Frames of Reference 


One cannot distinguish between norm- and criterion-referenced tests simply by 
looking at exemplars of each type. In fact, to a great extent, the distinctions be- 


Table 3.3 Expectancy Table: Relationship Between Midterm Test Scores 
and Final Grades 


Percentage Receiving Each 
Final Grade 


Midterm Test Score Number of Cases DorF C B A 
90 and above 9 11 22 67 
80 to 89 12 8 25 50 17 
70 to 79 13 23 46 31 

60 to 69 3 33 67 


59 and below 3 67 33 
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tween both frames of reference for test interpretation—as well as among the va- 
rieties within each frame—are matters of emphasis. Even though scores can be 
expressed in a variety of ways, fundamentally, all of testing relies on a normative 
framework. 

The standards used in criterion-referenced test score interpretations must be 
based on expectations that are realistic or feasible for the population of test tak- 
ers for whom the test is meant. Depending on the goals for which a test is used, 
it may be that too few or too many people are able to satisfy the criterion, in which 
case the test may prove to be impractical. In other words, criteria are based both 
on the testing purpose and also, to some extent, on what people can accomplish 
in a given situation. 

The use of cutoff scores or other standards of performance in criterion-refer- 
enced interpretation does not mean that differences in the test performance of 
individuals will be eliminated or go unnoticed, nor does it preclude score com- 
parisons across individuals. For instance, even if two applicants for a secretarial 
job meet the 65-words-per-minute criterion on a typing test, all other things be- 
ing equal, the one who can type 90 words per minute is more likely to be chosen 
over the one who can type only 65. 

Similarly, the use of norms does not prevent an examination of test perfor- 
mance from the point of view of content or behavioral criteria. In scoring class- 
room tests, for example, teachers who have carefully sampled from the material 
assigned for a test may choose to assign letter grades on the basis of the norms 
for the class (on the curve grading) or on the basis of performance criteria tied to 
the percentage of items answered correctly. Some standardized tests used in ed- 
ucational settings—such as the Iowa Tests of Basic Skills (TTBS), the Stanford 
Diagnostic Mathematics Test (SDMT), and the Stanford Diagnostic Reading 
Test (SDRT)—also try to provide for both norm-referenced and criterion-refer- 
enced interpretations. However, reviews of these tests suggest that this attempt 
meets with success only with regard to one type of interpretation or the other, but 
not both, because norm-referenced and criterion-referenced tests requite some- 
what different emphases in the way they are constructed. For more information 
on this issue, consult the reviews of the ITBS, SDMT, and SDRT in the 13th Men- 
tal Measurements Yearbook, edited by Impara and Plake (1998). 

How, then, does norm-referenced test interpretation differ from criterion-ref- 
erenced interpretation? The fundamental difference between the two is in their 
primary objectives: 


° In norm-referenced testing, the primary objective is to make distinc- 
tions among individuals in terms of the ability or trait assessed by a test; 


C ESSENTIALS OF TEST SCORE INTERPRETATION |13 


* Incriterion-referenced testing, the primary objective is to evaluate a 
person’s degree of competence or mastery of a skill or knowledge do- 
main in terms of a pteestablished standard of performance. 


As we have seen, these two objectives are not always or necessarily mutually ex- 
clusive and, in some situations, the same instrument can be used for both objec- 
tives. Which of the two objectives is the primary one is determined by the specific 
purpose of the test user. That purpose, in turn, should help the test user deter- 
mine which test development approach is the most suitable. 

With regard to the varieties of criterion-referenced test interpretation, it 
should be fairly clear that the distinction between domain-based and perfor- 
mance-based assessment is also arbitrary to some extent. Knowledge of content 
domains must be demonstrated through observable behavior ot performance in 
which one or more skills play a part. By the same token, every kind of perfor- 
mance requires some type of knowledge. Moreover, whereas in subjects that are 
elementary and fairly structured content domains can be mapped out, when it 
comes to more advanced and less structured areas—such as knowledge that cuts 
across various disciplines—this kind of parceling becomes difficult or impos- 
sible. Similarly, thresholds for mastery can be preestablished easily with regard to 
basic skills, but in fields that involve higher level skills, such standard setting may 
not be applicable because achievements are far more wide ranging, 


Criterion-Referenced Test Interpretation in Clinical Assessment 


Criteria such as mastery of a skill or knowledge in a given field are clearly not ap- 
plicable in connection with instruments designed to assess personality. There- 
fore, the term criterion-referenced interpretation is not ordinarily used for tests of this 
type. Nevertheless, some tests relevant to both emotional and cognitive func- 
tioning are used by clinicians to help establish whether certain diagnostic criteria 
have been met. These tests use cutoff scores—established on the basis of nor- 
mative data—to screen for the presence of certain disorders based on established 
links with clinical criteria. This particular application constitutes criterion-refer- 
enced interpretation, in the same sense as when the relationship between test 
scores and criteria is used to select or place individuals in educational or employ- 
ment settings. Both of these practices involve the estimation or prediction of cer- 
tain outcomes. Likewise, this sort of interpretation of clinical tests depends on va- 
lidity evidence relevant to the diagnostic criteria (see Chapter 5). Examples of 
tests that are used in this fashion include instruments such as the Beck Depres- 
sion Inventory, which can help to evaluate the intensity of depressive disorders, 
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and the Mini-Mental State Examination, which helps in screening for cognitive 
impairment. These tests and other clinical tools—such as behavioral checklists 
and rating scales—that involve the use of cutoffs and score ranges to evaluate be- 
havior symptomatic of mental disorders can also be said to be criterion-refer- 
enced. 


Item Response Theory As a Basis for Combining Frames of Reference 


Because their goal is to estimate a test taker’s position on a latent trait or ability 
dimension, IRT methods are well suited to the development of tests whose scores 
can be interpreted in terms of both normative and criterion-referenced bases. Al- 
though the data used in IRT modeling are derived from the performance of ref- 
erence samples, they can be combined with other kinds of item analyses to create 
scales that provide information of both a comparative (i.e., norm-referenced) and 
substantive (i.e., criterion-referenced) nature. One recent example of how this 
can be accomplished is found in the work Primi (2002) has done, which integrates 
cognitive theory and IRT in the construction of a measure of the type of ability 
needed to solve geometric matrix problems. Another is a hierarchical IRT model 
proposed by Janssen, Tuerlinckx, Meulders, and De Boeck (2000), which is ap- 
plied to a test measuring mastery targets in reading comprehension at the ele- 
mentary school level. 


Social Considerations in Norm- and Criterion-Referenced Testing 


The prevailing pressures to use criterion-referenced testing to certify competen- 
cies and make consequential decisions in educational and occupational settings 
has grown out of a dissatisfaction with the perceived weaknesses of norm-refer- 
enced testing. Some of this dissatisfaction atises from the view that the use of 
norm-referencing in education is a major cause of declining standards, given that 
no matter how poorly a student population might perform as a whole, relative to 
their own norms, at least half of them will always be above average. 

Another source of dissatisfaction with norm-referenced testing stems from 
the fact that, when used as a basis for decisions in the educational and occupa- 
tional realms, it often places members of underprivileged minority groups at a 
disadvantage compared to individuals who may have had more educational op- 
portunities. However, in an ironic twist, during the past few decades the use of 
ctiterion-referenced testing to certify competencies has become a subject of as 
much or more heated debate than norm-referenced testing in the professional 
and political arenas. Undoubtedly, much of the debate about both types of test- 
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ing hinges on misunderstandings of the proper role of tests as tools— rather than 
as arbiters—as well as on policies that seem to alternate between the two ex- 
tremes of overemphasis on and opposition to standardized testing in what have 
come to be known as “high stakes” decisions (see, e.g., Jaeger, 1989; Mehrens, 
1992; U.S. Department of Education, Office for Civil Rights, 2000; Wigdor & 
Green, 1991). 


elm TEST YOURSELF wA- 


|. If not accompanied by further information, a high raw score is 
a) meaningless. 
b) still always better than a low score. 


2; constitute the most widely used frame of reference for test 
score interpretation. 


a) Content domains 
b) Work samples 


c) Criteria 
d) Norms 


3. Of all the following developmental norms, which ones are the most uni- 
versally applicable? 


a) Theory-based ordinal scales 
b) Mental age norms 

c) Natural sequences 

d) Grade-based norms 


4. With regard to the samples used to establish within-group norms, the 
single most important requirement is that they should be 


a) gathered locally by the institution or organization that will use them. 
b) very large, numbering in the thousands. 

c) representative of the group for which they will be used. 

d) convenient to obtain in the process of standardization. 


5. The concepts of test ceiling and test floor are most closely related to the 
issue of 


a) test validity. 
b) test difficulty. 
c) the type of standard scores used in a test. 


d) the type of items used in a test. 


(continued ) 
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. When transformed into the Wechsler-scale type of deviation IQs, a 


z score of -1.00 would become a Wechsler IQ of 
a) 85. 

b) 95. 

©) 105. 

d) 115. 


Which of the following score transformation procedures is the only one 
that qualifies as a linear transformation? 


a) Normalized standard scores 
b) Percentiles to stanines 
c) Raw scores to percentile scores 


d) z scores to T scores 


. Ifa group of present-day individuals were to take both the original Wech- 


sler Adult Intelligence Scale (WAIS) and its latest revision, the WAIS-III, 
chances are that their IQ scores would be 


a) the same on both tests. 
b) higher on the WAIS-III than on the WAIS. 
c) higher on the WAIS than on the WAIS-III. 


The essential characteristic of item response theory models is that they 
place on a common scale. 


items and tests 


items and persons 


a 
b 
c) persons and tests 
(e) 


ne of the main advantages of IRT methodology is that it is ideally suited 
for use in testing. 


a) computer adaptive 
b) paperand-pencil 
c) personality 


Answers: ILa 2d; 3.64. c 5-b; 6.a; 7.d;8.¢9.b; 10.a. 


Four 


ESSENTIALS OF RELIABILITY 


he term re/iability suggests trustworthiness. To the extent that decisions of 

any kind are to be made, wholly or in part, on the basis of test scores, test 

users need to make sure that the scores are reasonably trustworthy. When 
used in connection with tests and measurements, reliability is based on the con- 
sistency and precision of the results of the measurement process. In order to have 
some degree of confidence or trust in scores, test users require evidence to the ef- 
fect that the scores obtained from tests would be consistent if the tests were re- 
peated on the same individuals or groups and that the scores are reasonably pre- 
cise. 

Whereas reliability in measurement implies consistency and precision, lack of 
reliability implies inconsistency and imprecision, both of which ate equated with 
measurement error. In the context of testing, measurement error may be defined as 
any fluctuation in scores that results from factors related to the measurement 
process that ate irrelevant to what is being measured. Reliability, then, is a quality 
of test scores that suggests they are sufficiently consistent and free from mea- 
sutement error to be useful. 

Note that, in order to be useful, test scores do not need to be either totally con- 
sistent ot error free. As we saw in Chapter 1, even in the physical sciences—some 
of which can boast of incredibly reliable instrumentation—measurements are al- 
ways subject to some degree error and fluctuation. In the social and behavioral 
sciences, measurements are much more prone to error due to the elusive nature 
of the constructs that ate assessed and to the fact the behavioral data through 
which they are assessed can be affected by many more intractable factors than 
other types of data (see Rapid Reference 5.2 on Deconstructing Constructs in 
Chapter 5). Psychological test scores, in particular, are especially susceptible to in- 
fluences from a variety of sources—including the test taker, the examiner, and 
the context in which testing takes place—all of which may result in variability that 
is extraneous to the purpose of the test. 
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TRUTH AND ERROR IN PSYCHOLOGICAL MEASUREMENT 


One of the most enduring approaches to the topic of reliability is the classical test 
theory notion of the zrue score (see, e.g., Gulliksen, 1950). In a way, this notion 
could be said to represent the object of the quest, or Holy Grail, of the psycho- 
metric enterprise. Although true scores do not really exist, it is nevertheless pos- 
sible to imagine their existence: True scores are the hypothetical entities that 
would result from error-free measurement. Methods for estimating the reliability 
of scores provide a way of estimating true scores, or at least the boundaries within 
which true scores might lie. The concepts of reliability and error in test scores— 
which must obviously enter into consideration with regard to any score—are 
applied in parallel yet somewhat different ways when dealing with one or more 
scores of a single individual than when dealing with the scores of groups. 


The Concept of the True Score in Individual Data 


In classical test theory, an individual’s rue score is conceptualized as the average 
score in a hypothetical distribution of scores that would be obtained if the indi- 
vidual took the same test an infinite number of times. In practice, it is obviously 
impossible to obtain such a score for even one individual, let alone for many. In- 
stead of true scores, what one derives from tests are observed scores (i.e., the scores 
that individuals actually obtain). 

With regard to a single score, the ideas presented so far can be stated succinctly 
by means of the following equation: 


<x (4.1) 


true error 


which expresses the concept that any observed score (X) is made up of two com- 
) and an error score component (X o4- 


a) 


ponents: a true score component (X u 
From a realistic point of view, the magnitudes of both of these components will 
always remain unknown in any given instance. Nevertheless, in theory, the true 
score component is construed to be that portion of the observed score which re- 
flects whatever ability, trait, or characteristic the test assesses. Conversely, the er- 
ror component, which is defined as the difference between the observed score 
and the true score, represents any other factors that may enter into the observed 


score as a consequence of the measurement process. 


True Scores in Group Data 


The singular importance of interindividual variability was already discussed in 
Chapter 2, where it was pointed out that the usefulness of psychological testing 
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hinges on obtaining some variability across individuals. Without score variability, 
tests could not help us to make comparative decisions about people. It may also 
be recalled that, in the same chapter, the sample variance (s*) was defined as the 
average amount of variability in a group of scores. Based on this information, 
Formula (4.1)—which pertains to a single test score—can be extrapolated and 
applied to the distribution of test scores obtained from a sample, or a population, 
in the following fashion: 


Sample variance = s? = 5? + s? (4.2) 
or 
Population variance = 6? = 0? + 0? (4.3) 


Both of these formulas express the same idea, namely, that the variance in a set of 
observed scores for a sample (s°) or a population (O°) consists of a portion that 
is true variance (s? or O?) and a portion that is error variance (s? or O°). True vari- 
ance consists of those differences among the scores of individuals within a group 
that reflect their standing or position in whatever characteristic the test assesses. 
Error variance is made up of the differences among test scores that reflect factors 
irrelevant to what the test assesses. Formulas (4.2) and (4.3) also imply that the re- 
liability of scores increases as the error component decreases. In fact, a reliability 
coefficient (r..)—about which more will be said later in this chapter—may be de- 
fined as the ratio of true score variance (s?) to total test score variance (s7), or, 


r == (4.4) 


In other words, if all the test score variance were true variance, score reliability 
would be perfect (1.00). A reliability coefficient may be viewed as a number that 
estimates the proportion of the variance in a group of test scores thatis accounted 
for by error stemming from one or more sources. From this perspective, the eval- 
uation of score reliability involves a two-step process that consists of (a) deter- 
mining what possible sources of error may enter into test scores and (b) estimat- 
ing the magnitude of those errors. 


THE RELATIVITY OF RELIABILITY 


Although the practice of describing zesżs as reliable is common, the fact is that the 
quality of reliability is one that, if present, belongs not to tests but to test scores. 
This distinction is emphasized consistently by most of the contributors to the 
contemporary literature on reliability (see, e.g, Rapid Reference 4.1). Even 
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though it may seem subtle at first 


= hpi Ra OJCIEC 4 f glance, the distinction is fundamental 
A Aa eaeaes = to an understanding of the implica- 
An excellent collection of readings on 
reliability, with more detailed and ex- ; 
tensive explanations of many topics regard to the use of tests and the in- 
covered in this chapter, can be found terpretation of test scores. If a test is 

in Score Reliability: Contemporary Think- 


ing on Reliability Issues, edited by Bruce i : ord 
Thompson (2003b). is that its reliability has been estab- 


tions of the concept of reliability with 


described as reliable, the implication 


lished permanently, in all respects, for 

all uses, and with all users. This would 
be akin to saying that a fine piano that is well tuned will always be in tune and will 
sound just as good regardless of the kind of music that is played on it or of who 
plays it. In fact, the quality of the sound a piano makes is a function not only of 
the instrument itself, but also of variables related to the music, the piano player, 
and the setting (e.g., the acoustics of the room) where the piano is played. Simi- 
larly, although reliability in testing hinges to a significant extent on characteristics 
of the test itself, the reliability of test scores—which is what results from the use 
of the instrument and, like the music a piano makes, is what really matters—can 
also be affected by many other variables. 

Even as applied to test scores, moreover, the quality of reliability is relative. 
The score a person obtains on a test is not reliable or unreliable in any absolute 
sense. Rather, an obtained score may be mote or less reliable due to factors 
uniquely pertinent to the test taker (e.g,, fatigue, lack of motivation, the influence 
of drugs, etc.) or to the conditions of the testing situation (e.g., the presence of 
distracting noises, the personality of the examiner, the strictness with which time 
limits are enforced, etc.). All of these factors may singly or jointly affect the ob- 
tained score to a greater or lesser extent, including up to the point where the score 
becomes so unreliable that it should be discarded. Even though they are unrelated 
to the test itself, all such matters need to be taken into account in the assessment 
process. 

In contrast, when reliability (7) is considered from the point of view of test 
score data obtained from a large sample, under standardized conditions, the measure- 
ment errors that may impinge on the individual scores of members of the sample, 
while still present, are assumed to be distributed at random. Since random errors 
are equally likely to influence scores in a positive or a negative direction, the er- 
rors can also be assumed to cancel each other out. Even then, however, teliabil- 
ity estimates will vary from sample to sample depending on their composition 
and the circumstances in which the testing occurs. For instance, if a test is aimed 
at adults ranging in age from 18 to 90 years, the estimated reliability of its scores 
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will be susceptible to the influence 


of different factors depending on DON “dl FORGET 


whether the estimate is based on data e Reliability is a characteristic of test 


obtained from the older age groups, scores, rather than of tests them- 

from the younger age groups, or from selves. 

a group that is representative of the e The reliability of any measurement, 

entire age range for which the test is and of psychological test scores in 

. particular is neither absolute nor 

intended. immutable. Thus, the possible 
sources of error in measurement, 
and the extent to which they enter 

A Note About Truth and Error into any specific use of a test, must 
be taken into account, estimated, 

The need to identify and investigate and reported every time test 

the true components and the error scores are employed (AERA, APA, 
NICME, 1999). 


components of scores is considered 
in more detail in the next section. At 
this point, however, it is important to emphasize that these judgments always 
have to be made in relation to what a test attempts to assess and to the circum- 
stances under which it is administered. For example, test scores that denote the 
speed with which individuals can place 100 pegs in a pegboard, if given under 
standardized conditions, would by and large reflect the test takers’ typical levels 
of manual dexterity quite reliably. If the same test were given (a) under conditions 
meant to distract the test takers or (b) under conditions that unintentionally dis- 
tracted the test takers, both sets of scores would reflect levels of manual dexter- 
ity under distracting conditions. However, the influence of the distractions would 
be seen as a source of error variance, or as reducing the reliability of what the 
scores are intended to indicate only in the second instance. 


SOURCES OF ERROR IN PSYCHOLOGICAL TESTING 


As we have seen, error can enter into the scores of psychological tests due to an 
enormous number of reasons, many of which are outside the purview of psycho- 
metric estimates of reliability. Generally speaking, however, the errors that enter 
into test scores may be categorized as stemming from one or more of the follow- 
ing three sources: (a) the context in which the testing takes place (including fac- 
tots telated to the test administrator, the test scorer, and the test environment, as 
well as the reasons the testing is undertaken), (b) the test taker, and (c) the test it- 
self. Some of the errors stemming from these sources can be minimized or elim- 
inated provided that proper testing practices are followed by the parties who are 
involved in the process of developing, selecting, administering, and scoring tests. 
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Still others, such as test takers’ carelessness or attempts at impression manage- 
ment in responding to test items, cannot be eliminated but may be detected by 
various types of checks built into the test. At any rate, practices related to the ap- 
propriate use of tests—practices most of which are aimed at reducing the error 
in scores—are discussed at greater length in Chapter 7, which deals with issues 
relevant to test selection, administration, and scoring, among others. 

For the purpose of this discussion of reliability, it will be assumed that test 
users, test administrators, and test scorers carefully select the appropriate instru- 
ments, prepare suitable test environments, establish good rapport with the test 
takers, and both administer and score tests in accordance with well established 
standardized procedures. Furthermore, it will also be assumed that test takers are 
properly prepared and well motivated to take tests. Whether these assumptions 
are ot are not justified in specific instances, the fact remains that the behaviors 
that they entail are subject to the control of one or more of the individuals in- 
volved in the testing process and are not pertinent to the tests themselves in a di- 
rect way. To the extent that these assumptions are justified, the error in test scores 
that stems from sources unrelated to the tests can obviously be eliminated or at 
least minimized. 

In considering reliability for the remainder of this chapter, the sources of er- 
ror to be discussed pertain primarily to factors outside the conscious control of 
the parties in the testing process, namely, random or chance factors. Before we 
proceed, however, it should be noted that measurement error can be systematic 
and consistent as well as random. Just as a balance may be off by a few pounds, a 
test may have an intrinsic characteristic of some sort that affects all test takers. 
Traditional estimates of reliability may fail to detect this sort of consistent error, 
depending on its source, because they are based on methods meant to detect in- 
consistencies in the results of a test. Systematic and consistent errors in mea- 
surement affect not only the reliability but also the validity of test results. In or- 
der to detect them, one must be able to compare the results of one instrument 
with those of other tools that assess the same construct but do not share the fac- 
tor that causes the consistent error. To detect the error in the case of the balance 
that is off by a few pounds, for instance, one would have to weigh the same per- 
son or object on one ot more additional and well calibrated scales. 

Rapid Reference 4.2 lists some of the possible sources of error that may ren- 
der test scores inconsistent. This list categorizes the sources of error assessed by 
traditional estimates of reliability, along with the types of tests to which those er- 
ror sources pertain most directly, and the reliability coefficients typically used to 
estimate them. A conceptual explanation of each of the sources of error and reli- 
ability estimates is presented next, in the same order as they ate listed in Rapid 
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Sources of Measurement Error With Typical Reliability 
Coefficients Used to Estimate Them 


Source of Error 


Type of Tests Prone 
to Each Error Source 


Appropriate 
Measures Used to 
Estimate Error 


Interscorer differences 
Time sampling error 


Content sampling error 


Interitem inconsistency 


Interitem inconsistency 
and content heterogeneity 
combined 


Time and content 
sampling error combined 


Tests scored with a de- 
gree of subjectivity 


Tests of relatively stable 
traits or behaviors 


Tests for which consis- 
tency of results, as a 
whole, is desired 


Tests that require inter- 
item consistency 


Tests that require inter- 
item consistency and 
homogeneity 


Tests that require stability 
and consistency of re- 
sults, as a whole 


Scorer reliability 


Test-retest reliability (r,,) 
a.k.a. stability coefficient 


Alternate-form reliability 
(nı) or split-half reliability 


Split-half reliability or 
more stringent internal 
consistency measures, 
such as Kuder 
Richardson 20 (K-R 20) 
or coefficient alpha (a) 


nternal consistency 
measures and additional 
evidence of homogene- 
ity 

Delayed alternate-form 
reliability 


Reference 4.2. Considerations regarding when and how these concepts and pro- 


cedures ate applied in the process of test use are discussed in a subsequent sec- 


tion of this chapter. 


Interscorer Differences 


Interscorer (ot interrater) differences is the label assigned to the errors that may enter 


into scores whenever the element of subjectivity plays a part in scoring a test. It 


is assumed that different judges will not always assign the same exact scores or 


ratings to a given test performance even if (a) the scoring directions specified in 


the test manual are explicit and detailed and (b) the scorers are conscientious in 


applying those directions. In other words, score variability that is due to inter- 
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scorer differences does not imply carelessness either in preparing the directions 
for scoring or in the actual scoring of a test. It refers to variations in scores that 
stem from differences in the subjective judgment of the scorers. 


Scorer Reliability 

The basic method for estimating error due to interscorer differences consists of 
having at least two different individuals score the same set of tests, so that for 
each test taker’s performance two or more independent scores are generated. The 
correlations between the sets of scores generated in this fashion are indexes of 
scorer reliability. Very high and positive correlations, in the order of .90 or higher, 
suggest that the proportion of error that is accounted for by interscorer differ- 
ences is 10% ot less, since 1 — @ .90) = < .10. 


Time Sampling Error 


Time sampling error refers to the variability inherent in test scores as a function of 
the fact that they are obtained at one point in time rather than at another. This 
concept hinges on two related notions, namely, (a) that whatever construct or be- 
havior a test evaluates is liable to fluctuate in time, and (b) that some of the con- 
structs and behaviors assessed through tests are either less subject to change, or 
change at a much slower pace, than others. For example, psychological constructs 
related to abilities, such as verbal comprehension or mechanical aptitude, usually 
are seen as being less prone to fluctuation over time than constructs related to 
personality, such as agreeableness or warmth. Within the realm of personality and 
emotional functioning the difference between more and less stable constructs 
has been codified in the traditional distinction of ¢raits—which are construed to 
be relatively enduring characteristics—versus states, which are by definition tem- 
porary conditions. To some extent, this distinction may also be applied to cogni- 
tive characteristics. Verbal ability, for instance, is assumed to be far more stable 
within an individual than attention and memoty capacities, both of which are 
more susceptible to the influence of transient conditions or emotional states. At 
any rate, it is clear that whereas a certain amount of time sampling error is as- 
sumed to enter into all test scores, as a rule, one should expect less of it in the 
scores of tests that assess relatively stable traits. 


Test-Retest Reliability 

To generate estimates of the amount of time sampling error liable to affect the 
scores of a given test, it is customary to administer the same test on two different 
occasions, separated by a certain time interval, to one or more groups of individ- 
uals. The correlation between the scores obtained from the two administrations 
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is a fest-retest reliability (or stability) coefficient (r) and may be viewed as an index of 
the extent to which scores are likely to fluctuate as a result of time sampling error. 
When this procedure is used, the time interval between the two test administra- 
tions always has to be specified, as it will obviously affect the stability of the 
scores. Realistically speaking, however, there are many factors that can differen- 
tially affect the test scores derived from a group of people across two occasions. 
Because of this, there is no fixed time interval that can be recommended for all 
tests. If the interval is very short, for instance, test takers may remember their re- 
sponses from the first occasion and this could affect their scores on the second 
one. On the other hand, if the interval is very long, there is always the possibility 
that intervening experiences—including steps that the test takers may have taken 
in reaction to the first administration—may affect the scores on the second oc- 
casion. In addition to these considerations, test users must also evaluate stability 
coefficients from the point of view of theoretical expectations pertinent to the 
traits and behaviors assessed by the test. One example would be the differences 
in the rate of change that may be expected as a function of the age of the test tak- 
ets. Reading comprehension, for instance, can change fairly rapidly in young chil- 
dren but should remain stable through adulthood, unless some unusual circum- 
stance—such as special training or brain injury—affects it. 


Content Sampling Error 


Content sampling error is the term used to label the trait-irrelevant variability that 
can enter into test scores as a result of fortuitous factors related to the content of 
the specific items included in a test. A simple example of how content sampling 
error might enter into test scores is presented in Rapid Reference 4.3. This illus- 
tration of content sampling error is rather contrived because it pertains to error 
introduced into test scores as a result of faulty test construction, and thus could 
have been avoided easily. In the example, the teacher’s selection of items results 
in inadequate coverage of the content knowledge the test is supposed to evaluate. 
Consequently, a good portion of score variability is unrelated to the students’ 
level of mastery of the specified material, rendering their scores not only less re- 
liable but also less valid than they would be otherwise. A more typical example is 
posed by cases in which—for reasons outside of the control of the test devel- 
oper—the specific content of a test either favors or disadvantages some of the 
test takers, based on their different experiential histories. For instance, a test de- 
signed to evaluate reading comprehension may fortuitously include several pas- 
sages that ate familiar to some test takers and not familiar to others. Obviously, 
those test takers who are familiar with the passages would be able to answer ques- 
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A Simple Illustration of Content Sampling Error Resulting 
From Faulty Test Construction 


Take the case of a content-referenced classroom test that is intended to evaluate 
knowledge of all the material contained in five textbook chapters. Suppose that 
the teacher preparing the test develops most of the items from the content of 
just three of the chapters, neglecting to include items from the remaining two 
chapters. Suppose further that several of the students have also concentrated on 
only three chapters in studying for the test. 


Content sampling error in the scores on such a test would result primarily from 
the teacher's uneven sampling from the material that the test was intended to 
cover. All other things being equal, the consequences of the teacher's inadequate 
content sampling would be as follows: (a) Those students who studied the same 
three chapters from which the test content was drawn would score close to 
100%, (b) those who concentrated on two of those chapters and one other 
would get grades of about 67%, and (c) those who were unfortunate enough to 
concentrate on only one of the “right” chapters, and the two chapters not in- 
cluded in the test, would score at approximately 33%. 


If we assumed that all the students who studied only three out of the five chap- 
ters had actually mastered 60% of the material the test was supposed to cover, 
their true scores should have approximated that percentage. The discrepancies 
between their obtained scores and their true level of mastery of the material is 
content sampling error. In this particular case, the error in the scores would lower 
not only the reliability of the scores but also their validity. 


tions based on them mote easily and expediently than the rest, on account of their 
greater familiarity with the material rather than because of a higher level of 
achievement in reading comprehension. 


Alternate-Form Reliability 

Alternate-form reliability procedures are intended to estimate the amount of error 
in test scores that is attributable to content sampling error. To investigate this 
kind of reliability, two or more different forms of the test—identical in purpose 
but differing in specific content—need to be prepared and administered to the 
same group of subjects. The test takers’ scores on each of the versions are then 
correlated to obtain alternate-form reliability (r,,) coefficients. Since it is unlikely that 
the same chance factors, favoring some test takers and not others, will affect the 
different forms, high and positive correlations (e.g., .90 or higher) between scores 
on the vatious forms may be taken as an indication that content sampling error is 
nota major influence on test scores (e.g., 10% or less). Delayed alternate-form re- 
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liability, a variation of this procedure 
used to assess the combined effects 
of time and content sampling, is dis- 
cussed later in this section. 


Split-Half Reliability 

Developing alternate forms of a test, 
ot administering the same test twice, 
often involves theoretical or practical 
problems that make these courses of 
action difficult. One solution is 
simply to administer a test to a group 
of individuals and to create two 
scores for each person by splitting the 
test into halves. 

Flow to split a test in half. The best way 
to split tests into halves for the pur- 
pose of computing split-half reliabil- 
ity coefficients depends on how tests 
were designed. In particular, it is im- 


DON’T FORGET 


The phrase all other things being equal 
appears in Rapid Reference 4.3 and 
elsewhere in this book. This phrase is a 
rhetorical device meant to suggest 
that all pertinent considerations, other 
than the concept under discussion at 
the time, are to be disregarded tem- 
porarily for the sake of isolating and 
clarifying the issue at hand. 


It is worth remembering, however 
that the assumption that “all other 
things” are equal is rarely, if ever, realis- 
tic in testing, as in other aspects of life. 
Instead, the phrase should serve (a) to 
alert the reader to the possibility that 
several other things do need to be 
considered, besides the specific con- 
cept under discussion, and (b) to stim- 
ulate the reader into pondering what 
those other things might be. 


perative to consider two possibilities: (a) whether some test items differ system- 
atically from other items across the length of the test, and (b) whether speed plays 
a significant role in test performance. Both of these conditions can have pro- 
found effects on the magnitude of split-half reliability coefficients. 


1. Systematic differences across test items can occur due to a variety of reasons. 


For example, many ability tests start with the easiest items and get pro- 


gressively more difficult, or are divided into parts or subtests that 
covet different content. Still others, like the Wonderlic Personnel Test, 
are structured in a spiral-omnibus format so that items dealing with ver- 


bal, numerical, spatial, and analytic tasks are rotated systematically. 


Many personality inventories are also arranged so that items from dif- 


ferent scales are rotated throughout the test. 


2. When test performance depends primarily on speed, items are usually pegged at 
a low enough difficulty level for all test takers to complete correctly, 


but time limits are set so that most test takers will not be able to finish 


the test. For example, tests of clerical aptitude often include tasks that 


require test takers to scan a long list of pairs of numbers, letters, or 


symbols within a brief time period and to indicate whether each pair is 
ot is not identical. In this kind of highly speeded test, scores depend 


128 ESSENTIALS OF PSYCHOLOGICAL TESTING ) 


primarily on the number of items completed, rather than on the num- 
ber of correct responses. Since most test takers will produce perfect or 
nearly perfect performance on all the items they attempt, any division 
of such a test into half-scores in terms of items—as well as any mea- 
sure of internal consistency—will yield nearly perfect coefficients. 


Rapid Reference 4.4 presents possible solutions to the problem of how tests of 
various types may be split into halves. Once this has been accomplished, the cor- 
relation between the scores on one half of the test and those on the other half (7) 
is used to derive a split-half reliability coefficient. Since r,, actually estimates the con- 
sistency of scores on the two half-tests, the Spearman-Brown (S-B) formulais applied 
to 7,,, to obtain the estimate for the full test. This formula is based on the classical 
test theory notion that a larger number of observations will produce a more reli- 
able result than a smaller number of observations. In other words, all other things 
being equal, a score that is based on a longer test will be closer to the true score 


than one based on a shorter test. The general version of the S-B formula is 
nr, 


XX 


eS 4.5 
GEETE a) 
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Some Solutions to the Problem of How to Split a Test in Half 


e A rule of thumb for splitting tests of various types to compute split-half reliabil- 
ity coefficients is to divide the test into the two most nearly comparable halves. Al- 
though this can be accomplished in many ways, it is often done by means of an 
odd-even split, with the odd items (1, 3, 5, etc.) and even items (2, 4, 6, etc.) 
making up the two halves. 


e To the extent that speed plays a role in test performance, any single-trial relia- 
bility estimate—such as the split-half method—will produce spuriously high re- 
sults. This occurs because, for tests that are significantly speeded, score reliabil- 

ity is primarily a function of the consistency of the speed with which test takers 

perform on the test, as opposed to the consistency of the caliber of their re- 
sponses. Thus, for speeded tests, one possible solution is to use two-trial reliabil- 
ity methods, such as test-retest or alternate forms. Another is to split the test in 
terms of separately timed halves and then compute the reliability coefficient in 
the same fashion as the usual split-half. 


e Why is this important to a potential test user? If the method used to compute es- 
timates of the internal consistency of test scores is not appropriate to the way 
a test was designed, the resulting reliability coefficients will be misleading. Po- 
tential test users who are considering the issue of reliability in the process of 
selecting a test need to attend to these matters. 
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where 


rs, = Speatman-Brown estimate of a reliability coefficient, 
n = the multiplier by which test length is to be increased or decreased, and 
r. = the reliability coefficient obtained with the original test length. 


This formula can be used to estimate the effect that lengthening a test by any 
amount, or shortening a test to any fraction of its original size, will have on the ob- 
tained coefficient. For example, if the obtained r for a 30-item test is 0.80 and one 
wishes to estimate what the reliability coefficient would be if the length of the test 
were increased to 90 items, by adding 60 comparable items, one would find to be 3 
and 7, ,, to be 0.92. If one wanted to shorten the same test to 15 items, # would be 
/Aand 7 p would go down to 0.67. When applied to a split-half reliability coefficient 
(n) Which involves estimating the reliability of the whole test based on the cor- 
relation between its two halves, the S-B formula can be simplified as follows: 


Age a (4.6) 


Interitem Inconsistency 


Interitem inconsistency refers to error in scores that results from fluctuations in ¿tems 
across an entire test, as opposed to the content sampling error emanating from 
the particular configuration of items included in the test as a whole. Although in- 
teritem inconsistencies may be apparent upon careful examination of item con- 
tent—and of the cognitive processes that may enter into play in responding to 
different items in a test—from the statistical point of view, they manifest them- 
selves in low correlations among test items. Such inconsistencies can be due to a 
variety of factors, including content sampling error, many of which are fortuitous 
and unpredictable. They can also result from content heterogeneity. 


Content Heterogeneity 


Content heterogeneity results from the inclusion of items or sets of items that tap 
content knowledge or psychological functions that differ from those tapped by 
other items in the same test. This factor is largely within the control of the test de- 
velopers, who should determine the degree of heterogeneity of test content based 
on the purpose of the test and on the kind of population for which the test is in- 
tended. To the extent that a test is purposefully designed to sample heteroge- 
neous content, content heterogeneity cannot be considered to be a source of 
error. Heterogeneity in test content, or in the cognitive functions tapped by 
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DON’T FORGET 


Deconstructing Heterogeneity and Homogeneity 


In psychological testing, the concepts of homogeneity and heterogeneity are used 
in reference to the composition of (a) the behavior samples or items that make 
up a test, and (b) groups of test takers, such as standardization samples or popu- 
lations. Since both of these aspects can affect all the statistics used to evaluate 
tests (see, e.g., the section on range restriction and correlation as well as Fig. 2.7 
in Chapter 2), it is important to remember the following: 


e Heterogeneity and homogeneity are always relative terms. Any entity that is com- 
posed of separate elements is heterogeneous to the extent that its elements 
are dissimilar in some respect. Thus, any group made up of multidimensional 
constituents, such as people or test items, is heterogeneous in some respects. 
By the same token, no such group is homogeneous in all respects. 


e In order to characterize a group as heterogeneous or homogeneous, it is necessary 
to decide which variable or variables will serve as the bases for evaluating simi- 
larity or dissimilarity. For instance: 


e The items in a test may be heterogeneous with regard to content and format— 
if some consist of words while others consist of numbers or if some are pre- 
sented orally, others in writing—but homogeneous with regard to cognitive 
function if they all involve memory (e.g., remembering words and numbers). 


e A group of people may be heterogeneous with regard to sex and age, if it in- 
cludes males and females ranging in age from |7 to 45, but homogeneous with 
regard to educational status, if it includes only college freshmen. 


different test items, is a source of error only when a test is supposed to be homo- 
geneous in one or more ways across all of its items. Rapid Reference 4.5 shows 
some item sets that vary in terms of their heterogeneity. 


Internal Consistency Measures 
Internal consistency measures ate statistical procedures designed to assess the extent 
of inconsistency across test items. Split-half reliability coefficients accomplish 
this to some extent. However, even a very short test can be split in half in several 
different ways—for example, a four-item test can be split into halves in 3 differ- 
ent ways, a six-item test in 10 ways, and so on—and each split may yield a some- 
what different correlation between the halves. One way to overcome this logisti- 
cal problem is do an odd-even split—with one half of the test consisting of odd 
items and the other half of even items—or any other split that will result in the 
two most nearly comparable halves (see Rapid Reference 4.4). 

Another solution is provided by formulas that take into account interitem cor- 
relation (i.e., the correlation between performance on al the items within a test). 
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Examples of Sets of Test Items From Most to Least 
Heterogeneous Content 


Set (A) 
ltem l: What number should come next in the following series? 
3 6 12 24 


Item 2: Which of the five ttems listed is least like the other four? 
Pork Beef Chicken Tuna Veal 


Item 3: A train travels 40 feet in 1/2 of a second. At this same speed, how far will 
it travel in 4 seconds? 


Set (B) 
ltem 1: 4+ |O= 
Item 2: Ifa dozen eggs cost $1.03, how much will three dozen eggs cost? 


ltem 3: The price of an item is reduced by 60% during a sale. By what percent 
should the price be increased to go back to the original price? 
60% 80% 100% | 20% 150% 

Set (C) 

ltem l: 4x5 = 

lem 2: 7x I| = 

ltem 3: 15x I5 = 


e Set A is the most heterogeneous: The items differ in terms of content domains, 
formats, and skills required. 


e Set B is next: The items share the same content domain (Math), but differ in for- 
mat and skills required (i.e., addition, multiplication, and fractions in Math plus 
basic reading skills). 

e Set C is the most homogeneous: Its items share a common domain, format, and 
skill required (understanding the operation of multiplication and its symbols). 


The two most frequently used formulas used to calculate interitem consistency 
are the Kuder-Richardson formula 20 (K-R 20) and coefficient alpha (Q), also known as 
Cronbach’s alpha (Cronbach, 1951), which is simply a more general case of the 
K-R 20 postulation. Both the K-R 20 and coefficient alpha require a single (one- 
trial) administration of a test to a group of individuals. The magnitude of the 
K-R 20 and alpha coefficients is a function of two factors: (a) the number of items 
in the test, and (b) the ratio of variability in test takers’ performance across all the 
items in the test to total test score variance. All other things being equal, the mag- 
nitude of K-R 20 and coefficient alpha will be higher (a) as the number of items 
increases and (b) as the ratio of item score variance to total test score variance de- 
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creases. Conceptually, both K-R 20 and coefficient alpha produce estimates of re- 
liability that are equivalent to the average of all the possible split-half coefficients 
that would result from all the possible different ways of splitting the test in half. 
As such, they represent a combined estimate of content sampling error as well as 
content heterogeneity. Therefore, unless a test is highly homogeneous, K-R 20 
and coefficient alpha reliabilities will be lower than any single split-half coeffi- 
cient. Rapid Reference 4.6 contains the K-R 20 formula and one version of the 
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Formulas for Calculating Internal Consistency 


Kuder-Richardson formula 20 (K-R 20) 


n \ s-2 pq 
NR 20 — a= 2 


t 
where 

n = number of items in the test 

s? = variance of total scores on the test 

x pq = sum of p times q for each item on the test 

p = proportion of persons who pass each item or answer it in a specific direction 


q = proportion of persons who fail each item or answer it in the opposite direc- 
tion 


The rera formula is applied to tests whose items are scored as right or wrong, or 
in any other dichotomous fashion, such as true or false, if all the items are phrased 
so that the meaning of each alternative is uniform throughout the test. 


Coefficient alpha (a) or Cronbach’s alpha 


n_\se-X(s)/) 
S| | ea 
a= | s? 


n = number of items in the test 


where 


s? = variance of total scores on the test 
X(s?) = sum of the variances of item scores 


This coefficient alpha formula and a variation known as standardized item alpha, 
which uses the average interitem correlation instead of item score and total score 
variances, are used for tests whose items have multiple possible responses (e.g, 
strongly agree, agree, disagree, and strongly disagree). Cortina (1993) provides an 
extensive discussion of the meaning of coefficient alpha formulas and the various 
factors that can affect their results. 
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coefficient alpha formula, along with a basic explanation of their components 
and applicability. 

Since both K-R 20 and coefficient alpha are heavily dependent on the amount 
of interitem variability within a test, it stands to reason that any lack of uniformity 
in the content of test items, such as content heterogeneity, will lower these coef- 
ficients. For instance, suppose internal consistency coefficients for three tests of 
equal leneth—made up of items like those presented in Sets A, B, and C of Rapid 
Reference 4.5—were calculated. If this were done, the test resembling the mix of 
items in Set A in terms of heterogeneity would have the lowest internal consis- 
tency because the differences in test takers’ mastery of the various skills and con- 
tent domains tapped by the test items would be reflected in their performances. 
The test with the most homogeneous items, namely, items such as those in Set C, 
would have the highest coefficient. 

Why is this important? When a test is purposefully designed to include items that 
are diverse in terms of one or more dimensions, K-R 20 and coefficient alpha will 
overestimate content sampling error, and thus are inappropriate. Depending on 
the design of the test, and based on an examination of its content, homogeneous 
items may be placed into subtests or otherwise segregated in order to compute 
separate measures of interitem consistency among groups of similar items. On 
the other hand, when homogeneity across all test items is desired, the magnitude 
of K-R 20 or coefficient alpha is an index of the extent to which this aim has been 
realized. In fact, the difference between the magnitude of the most appropriate 
split-half reliability coefficient and either the K-R 20 or Œ coefficients can be 
taken as an indication of the amount of heterogeneity in the items of a test. The 
closer the two estimates are, the more homogeneous the test content is. 

Factor analytic techniques can also be used to investigate the heterogeneity 
and possible multidimensionality of test items. These techniques, discussed at 
greater length in Chapter 5, are used to detect similarities among a set of vari- 
ables—such as responses to test items—based on the interrelatedness of their 
patterns of variability among one or more groups of test takers. 


Time and Content Sampling Error Combined 


Time sampling and content sampling error can be estimated in a combined fash- 
ion for tests that require both stability and consistency of results. As we shall see 
shortly, it is also possible to estimate the combined effects of other sources of er- 
ror on test scores through other means. However, the delayed alternate-form de- 
sign provides a good method for estimating time and content sampling error with 
a single coefficient. 
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Delayed Alternate-Form Reliability 

Delayed alternate-form reliability coefficients can be calculated when two or more al- 
ternate forms of the same test are administered on two different occasions, sep- 
arated by a certain time interval, to one or more groups of individuals. Just as with 
test-retest reliability, the interval between the two administrations needs to be 
clearly specified, along with the make-up of the samples and other conditions that 
might affect the magnitude of the obtained coefficients. If the two forms of a test 
are administered in immediate or close succession, the resulting alternate-form co- 
efficient will be primarily a function of the reliability across forms. With longer in- 
tervals between the administrations of the two forms, the error variance in scores 
will reflect time fluctuations as well as content sampling error in the test. 

A note about practice effects. One inevitable consequence of using the same test, or 
alternate forms of a test, repeatedly with the same subjects is that this introduces 
an additional source of unwanted variability in scores due to practice effects. Natu- 
rally, the length of the interval between the administrations affects the extent to 
which scores on the second or subsequent administrations of the test will be sub- 
ject to practice effects. With short intervals, practice effects can be quite signifi- 
cant, especially when test items involve novel tasks that require test takers to 
grasp certain problem-solving strategies likely to be remembered. One-trial 
methods for estimating score reliability, such as the split-half technique and co- 
efficient alpha, are not prone to practice effects, whereas two-trial procedures, 
such as test-retest and alternate-form reliability, usually are. To the extent that in- 
dividuals differ in the amount of improvement shown upon retesting due to prac- 
tice, the obtained correlations across two trials are likely to be reduced. Even 
more importantly, however, when tests are administered repeatedly for the pur- 
pose of assessing change over time, as they are in longitudinal studies, practice ef- 
fects can be a significant confounding variable that must be taken into account 
(see, e.g., Kaufman & Lichtenberger, 2002, pp. 163—165). 


RELIABILITY IN TEST USE 


Score reliability is a perennial consideration in psychological testing because of 
the ever present possibility that errors from various sources will enter into test re- 
sults. However, the manner in which the reliability of scores is considered differs 
at various points in the process of test development as well as in the actual appli- 
cation of tests. From the perspective of a test user, which is most pertinent for our 
purposes, reliability estimates must be carefully considered and applied in the 
stages of (a) test selection and (b) test score interpretation. Chapter 7 deals with 
these matters in greater detail; nevertheless, since the uses of statistical estimates 
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of reliability are presented in this chapter, the different ways in which reliability is 
considered at each stage will be introduced at this point to provide a context for 
the ensuing discussion. 


Reliability Considerations in Test Selection 


When test users are deciding which test to use for a given purpose, they must look 
at the data that have already been gathered concerning the reliability of the scores 
from specific tests. These data usually can be found in test manuals, handbooks, 
and articles prepared by the authors or developers of tests, but they may also ap- 
pear in the psychological literature as a result of the work of independent investi- 
gators who have used the tests. Typically, reliability data are presented in the form 
of correlation coefficients. Because of the pervasive use of the Pearson r coeffi- 
cient in evaluating the reliability and validity of test scores, the essential aspects of 
this correlational method, including its limitations (discussed in Chapter 2), must 
be thoroughly understood before delving into these topics. One particularly rel- 
evant fact to keep in mind is that the magnitude of correlation coefficients de- 
pends to some extent on the variability of the samples for which they were calcu- 
lated (see the section on Range Restriction and Correlation in Chapter 2). 

The various types of coefficients that can be computed to estimate measure- 
ment error, along with the most pertinent sources of error, have already been de- 
scribed, albeit in the abstract. At the point of selecting a test, potential test users 
need to apply these notions to the particular situations for which they wish to em- 
ploy a test. Rapid Reference 4.7 lists the basic steps involved in test selection from 
the point of view of reliability; a more extensive discussion of these considera- 
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Reliability Considerations in Test Selection 


Step | Determine the potential sources of error that may enter into the scores of 
the prospective instruments that are under review. 

Step 2 Examine the reliability data available on those instruments, including the 
types of samples on which the data were obtained. 


Step 3 Evaluate the data on reliability in light of all the other attributes of the tests 
in question, such as normative and validity data, cost and time constraints, 
and so forth. 


Step 4 All other things being equal, select the test that promises to produce the 
most reliable scores for the purpose and population at hand. 


136 ESSENTIALS OF PSYCHOLOGICAL TESTING +) 


tions is presented in the next few paragraphs. As a preliminary caveat, it must be 
noted that in evaluating the psychometric characteristics of a test—whether it be 
in relation to reliability, validity, normative data, or any other technical aspect of 
an instrument—there are no fixed rules that apply to all tests or all test uses. 


Evaluating Potential Sources of Error in Test Scores 

The foremost precaution for minimizing error in test scores is to adhere strictly 
to standardized procedures for the administration and scoring of tests (see Chap- 
ter 7). Beyond that, test users need to evaluate the possible relevance of each of 
the sources of error listed in Rapid Reference 4.2, in view of the choice of instru- 
ments available and of the purposes for which they might be employed. For ex- 
ample: 


° Ifthe scoring of a test involves subjective judgment, scorer reliability 
has to be taken into account. 

e Ifa test is going to be used to evaluate change over time, such as pos- 
sible improvement through a therapeutic intervention, an estimate of 
time sampling error—as well as of possible practice effects—in the 
scores of the instruments under consideration is essential. 

e If there is a possibility that a person will have to be retested at a later 
date to confirm or ratify previous findings, the availability of an alter- 
nate form of the test, with high delayed alternate-form score reliability, 
would be highly desirable. 

° If homogeneity and consistency across the entire test are desired, one 
would look for a high K-R 20 or alpha coefficient. 


Evaluating Reliability Data 
Reliability coefficients provide test users with some information concerning the 
magnitude of error that is likely to enter into scores from various sources. How- 
ever, in evaluating reliability data, one must keep in mind the fact that these esti- 
mates are affected by the characteristics of the sample for which they were com- 
puted and may or may not generalize to other groups of test takers. Among other 
things, this means that small differences in the magnitude of coefficients of dif- 
ferent tests are not likely to be of as much significance as other considerations. 
Furthermore, in light of the variety of factors that can impinge on the reliability 
of test scores, there is a growing recognition that investigators must routinely in- 
clude score reliability data for their own samples in reporting the results of their 
research studies (see, e.g., Baugh, 2003; Onwuegbuzie & Daniel, 2002). 

When a test is to be used in individual assessment, as opposed to research in- 
volving group data, the importance of critically examining the published infor- 
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mation on score reliability prior to selecting a test cannot be overemphasized. In 
addition to evaluating the samples on which the data were obtained with regard 
to size, representativeness, and variability, test users should ponder whether the 
available coefficients are the most appropriate for the type of instrument at hand 
and for the intended uses of the test. Furthermore, if a test is made up of subtests 
or other parts whose scores are to be interpreted singly or in combination, esti- 
mates of reliability for each of the part scores should be available in addition to 
the estimates for the total test score. 

In a manner of speaking, a reliability coefficient might be described as the cor- 
relation of the test with itself. Though not totally accurate, this description is a re- 
minder that reliability coefficients are based on data—such as two administra- 
tions of the same test, two versions of the same test, interitem correlations, and 
so on— that ought to be highly consistent. Low reliability estimates (below .70) sug- 
gest that the scores one derives from a test may not be very trustworthy. Thus, al- 
though there is no minimum threshold for a reliability coefficient to be consid- 
ered as adequate for all purposes, it is understood that, all other things being 
equal, the higher the coefficient, the better. Most test users look for coefficients 
to be at least in the range of .80 or higher. 


Evaluating Score Reliability Data in Light of Other Test Attributes 

Test selection decisions must be made on a case-by-case basis, taking into account 
all the characteristics of the available instruments and of the constructs they pur- 
port to assess, as well as the requirements of the specific situation in which test 
scores will be used. As fundamental as score reliability is, it is by no means the 
only consideration in test selection. In addition to the issue of reliability, validity 
data (Chapter 5) and the availability of normative or criterion-referenced infor- 
mation for test score interpretation (Chapter 3) are of paramount importance. Al- 
though practical considerations—such as cost, ease of administration and scor- 
ing, time constraints, and the like—necessarily play a role in test selection, to the 
extent that test use is likely to have a significant impact on test takers, such con- 
siderations should not be the determining factors in test choice. 


Evaluation of Error From Multiple Sources 


Most test scores ate susceptible to measurement error stemming from more than 
one soutce. In classical test theory, this realistic possibility is accommodated by 
(a) methods that estimate the combined influence of two sources, such as delayed 
alternate-form reliability, which estimates both time and content sampling error; 
ot (b) adding up the amounts of error variance estimated by all pertinent reliabil- 
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ity coefficients to arrive at an estimate of total error variance. Both of these strate- 
gies hinge on the fact that reliability coefficients may be interpreted as estimates 
of the proportion of score variance attributable to error from various sources (see 
Formula [4.4]). For example, if the delayed alternate-form reliability coefficient 
ofa test is .75, 75% of score variance can be interpreted as true variance and 25% 
(1 —.75 = .25) of the variance can be attributed to the combined influence of time 
and content sampling error. If test scores are likely to be affected by several 
soutces of error, the reliability estimates that assess error from different sources 
may be combined. Rapid Reference 4.8 describes such an analysis of sources of 
error variance for scores from the WAIS-III Vocabulary subtest (Psychological 
Corporation, 1997). 


Generalizability Theory 

An alternative approach to reliability that attempts to be more comprehensive 
than the one we have been discussing is what has come to be known as generaliz- 
ability theory, ot simply G theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972; 
Shavelson & Webb, 1991). Generalizability theory is an extension of classical 
test theory that uses analysis of variance (ANOVA) methods to evaluate the 
combined effects of multiple sources of error variance on test scores simulta- 
neously. 

A distinct advantage that G theory has—compared to the method for com- 
bining reliability estimates illustrated in Rapid Reference 4.8—is that it also al- 
lows for the evaluation of the interaction effects from different types of error 
soutces. Thus, it is a mote thorough procedure for identifying the error variance 
component that may enter into scores. On the other hand, in order to apply the 
experimental designs that G theory requites, it is necessary to obtain multiple ob- 
servations for the same group of individuals on all the independent variables that 
might contribute to error variance on a given test (e.g,, scores across occasions, 
across scorers, across alternate forms, etc.). On the whole, however, when this is 
feasible, the results provide a better estimate of score reliability than the ap- 
proaches described earlier. In spite of the fact that G theory was originally intro- 
duced in the early 1960s, relatively few test authors have applied it in developing 
new instruments. However, as familiarity with this technique becomes more 
widespread, it is likely to gain in popularity. Readers who wish to become ac- 
quainted with the basic procedures of G theory might consult a brief introduc- 
tion provided by Thompson (2003a), which includes a simple computational 
example. A mote comprehensive and detailed treatment of the conceptual 
framework and statistical aspects of generalizability theory can be found in 
Robert Brennan’s volume on this topic (2001). 
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Analysis of Multiple Sources of Error Variance in 
Scores From a Single Test 


The Wechsler Adult Intelligence Scale—Third Edition (WAIS-II]) Vocabulary sub- 
test consists of a series of increasingly di 


taker by the examiner and simultaneous 


ficult words that are read to the test 
y presented visually in a stimulus booklet. 


The test taker’s definitions, given orally, are recorded verbatim and immediately 
scored by the examiner, using a scale of 2, |, or O points, depending on the quality 


of the examinee’s responses. In scoring the answers, examin 
thorough familiarity with samples of responses provided in the manual for each of 
the words—at the three score levels—as well as by the dic 


each word (Psychological Corporation, 


ers are guided by a 


tionary definitions of 


997). 


The total score for the Vocabulary subtest is the sum of the points earned by the 
examinee on all the attempted items (words). A score of this nature is susceptible 
to time and content sampling error, as well as to the possibility of interscorer dif- 
ferences. Average reliability estimates provided in the WAIS-III manual (which 
might be consulted by those who want more detailed information) for the Vocab- 


ulary subtest are as follows: 


Proportion and 


Error Source/ Average Percent (%) of 
Type of Reliability Coefficient Error Variance 
Time sampling/stability 

(test-retest) Al | —.91 = .09 (9%) 
Content sampling/internal 

consistency 93 | —.93 = .07 (7%) 
Interscorer differences/ 

interrater (scorer) 25 | —.95 = .05 (5%) 


Total measured error 
variance 


Estimated true variance 


09 + 07 + .05 = .21 (21%) 
| —.21 =.79 (79%) 


From the preceding calculations, it 


shou 


source estimate of reliability for a test of 
subtest of the WAIS-III would give a highly misleading impression of the possible 
amount of error in its scores. Furthermore, this example points out that in order 


for scores that are subject to multi 


thy the reliability estimates for eac 
range of .90 or higher. 


See Example |: Applying the SEM i 


h sing 


d be quite evident that to rely on a single- 
the kind exemplified by the Vocabulary 


ple sources of error to be sufficiently trustwor- 


e source needs to be quite high, in the 


n the 


text for a specific application of this anal- 


ysis of multiple sources of error and tts effect on the reliability of a score from the 


WAIS-III Vocabulary subtest. 
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The Item Response Theory Approach to Reliability 


Mote sophisticated methods of estimating reliability are available through item re- 
sponse theory (IRT) (introduced in Chapter 3 and discussed further in Chapter 6). 
A full explanation of the technical aspects of IRT models is beyond the scope of 
this text, but the advantages that these models provide, especially for large scale 
and computer adaptive testing, have been rapidly spurring their development and 
application in the past few decades. With IRT methods, reliability and measure- 
ment error are approached from the point of view of the information function of 
individual test items, as opposed to the test as a whole. Because the difficulty level 
and discriminative power of individual items—trelative to the trait assessed by the 
test—can be more carefully calibrated through IRT methods, the information that 
each test taker’s response provides is more precise and thus more reliable. In the 
type of computer adaptive testing that these methods allow, the selection of the 
most appropriate items to present to test takers is determined by their previous re- 
sponses. Using IRT methodology and adaptive testing, adequate reliability with 
minimal measurement error can be obtained with tests that are shorter than the 
traditional tests (which provide the same fixed content to all test takers), provided 
that a sufficiently extensive and inclusive item bank is available. This is just one of 
the many fundamental ways in which the model-based version of measurement 
known as IRT differs from the rules and assumptions of classical test theory (see, 
e.g., Embretson & Reise, 2000, pp. 13-39). 


RELIABILITY CONSIDERATIONS IN TEST INTERPRETATION 


Once a test has been chosen, administered, and scored, reliability data are applied 
in the process of test interpretation for two distinct but related purposes. The first 
is to acknowledge and quantify the margin of error in obtained test scores. The 
second purpose is to evaluate the statistical significance of the difference between 
obtained scores to help determine the import of those differences in terms of 
what the scores represent. 


Quantifying Error in Test Scores: The Standard Error of Measurement 


In the interpretation of any single score—or score average—from a test, reliabil- 
ity data are used to derive the upper and lower limits of the range within which test 
takers’ true scores ate likely to fall. A confidence interval is calculated for an ob- 
tained score on the basis of the estimated reliability of scores from the test in ques- 
tion. The size of the interval depends on the level of probability that is chosen. 
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Example 1: Applying the SEM 
The estimated score reliability of the Vocabulary subtest of the WAIS-III de- 
scribed in Rapid Reference 4.8—after subtracting the estimated error variance 
from three relevant sources—is .79. As is true of all Wechsler subtest scores, Vo- 
cabulary has scaled scores that can range from 1 to 19, with M = 10 and SD = 3. 
To illustrate the most basic application of reliability data, let us assume that an in- 
dividual named Maria obtains a score of 15 on the WAIS-III Vocabulary subtest. 
Step 1. In order to obtain a confidence interval for Maria’s obtained score (X) 
of 15, we need the standard error of measurement (SEM) for Vocabulary. The SEM is 
a statistic that represents the standard deviation of the hypothetical distribution 
we would have if Maria were to take this subtest an infinite number of times. As 
mentioned earlier in this chapter, the mean of such a hypothetical distribution 
would be Maria’s zrue score on the Vocabulary subtest. Inspection of the formula 
in Rapid Reference 4.9 reveals that the SEM is a function of the reliability coeffi- 
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Standard Error of Measurement (SEM) Formula 
SEM = SD VI =r, 


where 
SD, = the standard deviation of the test 
r = the reliability coefficient 


Standard Error of the Difference Between Two Scores 
(SE...) Formulas 


SE iq Formula l: 

S2 = SDV 2h) a 
where 
SD = the standard deviation of Test | and Test 2 


rı = the reliability estimate for scores on Test | 


fy = the reliability estimate for scores on Test 2 


SE jiy Formula 2: 

SE gig = V (SEM,)? + (SEM,)? 
where 
SEM, = the standard error of measurement of Test | 


SEM, = the standard error of measurement of Test 2 


SE sie Formula | is used if the two test scores being compared are expressed in 


the same scale, and SE gẹ Formula 2 is used when they are not. 
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cient of the scores of the test in question and that it is expressed in terms of the 
test’s standard deviation unit. Because of this, the size of the SEM cannot be taken 
by itself as an index of reliability. Tests that have large standard deviation units, 
such as the SAT with SD = 100, will have much larger SEMs than tests with small 
standard deviation units, such as the Wechsler scale subtests with SD = 3, even if 
their reliability coefficients are equal in magnitude. 

Step 2. Since we cannot ever obtain multiple Vocabulary subtest scores for 
Maria, nor average them to find an estimate of her true score, we must choose an 
available score that can be placed at the center of the interval to be created by the 
SEM. Here, two possibilities arise: (a) we can either use the obtained score, X,, as 
an estimate of Maria’s true score, or (b) we can estimate her true score (7”) with 
the following formula, based on Dudek (1979): 


T'=r (X, -M)+M (4.7) 
where 


T’ = the individual’s estimated true score 

r _ = estimated reliability of test scores 

X, = the individual’s obtained score 

M = the mean of the test score distribution 


In Maria’s case, since X, = 15, 7, = .79, and M = 10 for the Vocabulary subtest, 
as for all Wechsler subtest scores, her estimated true score is 14 (7” = (.79) (15 — 
10) + 10 = 13.95, or 14). Note that since her score is above the mean, Ler estimated 
true score is lower than her obtained score. In contrast, an obtained score of 5 on the 
same subtest—which deviates from the mean by an equal amount as Maria’s, but 
in the opposite direction—would result in an estimated true score of 6, which is 
higher than X, (if X, = 5, T” = (.79) (5-10) + 10 = 6.05, or 6). The reason for this 
difference in the true score estimates of obtained scores that are above the mean 


and those that are below the mean is that the estimation procedure takes into ac- 
count the effect of regression toward the mean. By the same token, if X, =M, the 
best estimate of the true score would be the mean itself. 

Step 3. Whether it is necessary to calculate 7” in order to create a confidence 
interval depends on how much an obtained score deviates from the mean. If an 
obtained score is close to the mean, the estimated true score will not differ from 
it by much; on the other hand, as obtained scores get more extreme, calculating 
estimated true scores that are closer to the mean becomes more advisable. Be that 
as it may, Step 3 involves calculating the SEM. Using the formula in Rapid Refer- 
ence 4.9, we find that SEM = 3\/1 — .79 = 1.37. Since this SEM, like the other 
standard errors described in Chapter 2, represents the standard deviation of a hy- 
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pothetical distribution of scores that is assumed to be normal, we can interpret it 
in terms of the normal curve frequencies. It may be recalled, from Chapter 3, that 
approximately 68% of the area under the normal curve is comprised within +1 
SD from the mean, 95% is within +1.96 SDs, and so on. Applying these percent- 
ages to Maria’s estimated true score (7) of 14 and applying the obtained SEM of 
1.37, we can say that (a) chances are 68/100, or p = .32, that Maria’s true score is 
somewhere within the interval of 14 + 1.37, that is between 13 and 15; and (b) 
chances are 95/100, or p = .05, that her true score is within 14 + (1.37) (1.96), that 
is, between 11 and 17. 


Interpreting the Significance of Differences Between Scores 


Assessment goals often entail comparisons (a) between two or more scores ob- 
tained by the same individual on different parts of a test battery, as when levels of 
performance in different domains are compared, or (b) between the scores of 
two or mote persons on the same test, for the purpose of evaluating their relative 
merits or characteristics. In both of these cases, reliability data may be used to de- 
rive probability statements concerning the likelihood that the obtained differ- 
ences between scores—and what the scores represent—could have been due to 
chance. The statistic used for this purpose is the standard error of the difference between 
scores, or SE jcc. 
Rapid Reference 4.9, depending on whether the scores to be compared are ex- 


It can be calculated using either one of the two formulas listed in 


pressed on the same scale (Formula 1) or not (Formula 2). Regardless of which 
of the two formulas is used the $E gwill be larger than the SEM of either one of 
the scores involved in the comparison because the evaluation of differences be- 


tween scores has to take into account the error present in both scores. 


Example 2: Applying the SE jg 

To illustrate the use of the standard error of the difference between scores, let us 
suppose we wish to estimate the statistical significance of the difference between 
Maria’s obtained scores on two subtests of the WAIS-III: her Vocabulary subtest 
score of 15 and her Information subtest score of 10. The Vocabulary subtest is 
described in Rapid Reference 4.8; the Information subtest assesses knowledge of 
common events, objects, places, and people. 

Step 1. Since we want to estimate the significance of a difference between two ob- 
tained scores, the first step is to calculate that difference. In this case, 15 — 10 = 5. 
There is a 5-point difference between Maria’s Vocabulary subtest score and her In- 
formation subtest score. Knowing this, we can proceed to evaluate whether the ob- 
tained difference is statistically significant (i.e., unlikely to have occurred by chance). 
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Step 2. We need to calculate the standard error of the difference between scores 
on the Vocabulary and Information subtests of the WAIS-III. Since the two sub- 
tests are expressed on the same score scale (M= 10 and SD = 3), we may use For- 
mula 1 from Rapid Reference 4.9. This requires knowledge of the reliability co- 
efficients for the subtest scores. The combined coefficient for Vocabulary is .79, 
as estimated in Rapid Reference 4.8. For the Information subtest, the coefficient 
is estimated to be .85, based on the combined internal consistency and stability 
coefficients of .91 and .94, respectively, available in the WAZS-II/ WMS-IL Tech- 
nical Manual (Psychological Corporation, 1997). Thus, the standard error of the 
difference between Vocabulary and Information subtest scores is 


SE e = 3V2- 79 = 85 = 1.80 


Step 3. To determine the statistical significance of the obtained score difference 
of 5 points, we divide that difference by the SE e 
5/1.80 = 2.78. 

Step 4. Consulting the Table of Areas of the Normal Curve in Appendix C, for 
az value of 2.78, we find that the area in the smaller portion that is cut off by that 
z value is .0027. Since there was no reason to presuppose that either one of the 
subtest scores (Vocabulary or Information) would be higher than the other, a 
two-tailed significance test for the null hypothesis of no difference between the 
scores is appropriate. Therefore, we multiply .0027 by 2 and obtain .0054, which 
indicates that the probability that Maria’s Vocabulary and Information subtest 
scores would differ by 5 points due to chance is 5.4 in 1,000. Given this high level 
of significance for the difference, all other things being equal, we can safely infer 


and obtain a critical value of 


that there really is a difference: Maria’s knowledge of vocabulary, as measured by 
the Vocabulary subtest of the WAIS-III, most likely exceeds her knowledge of 
general information concerning common events, places, objects, and people, as 
measured by the Information subtest. 

Why is it important to create confidence intervals for obtained scores and for differences be- 
tween obtained scores? Two basic reasons can be adduced in answer to this question. 
The first is that confidence intervals for obtained scores remind us that test scores 
are not as precise as theit numerical nature would seem to suggest. Thus, when- 
ever important decisions are to be made with the help of test scores, especially 
when cutoff scores are used, serious consideration has to be given to measure- 
ment error as quantified by the SEM. The second reason, which is related to the 
first, is that confidence intervals prevent us from attaching undue meaning to 
score differences that may be insignificant in light of measurement error. In 
recognition of the importance of these facts, many test manuals include tables 
listing the standard errors of measurement for scores as well as the numerical 
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ranges for each possible score that can be derived from a test, along with the lev- 
els of confidence for each score range. For instance, for an obtained Full Scale IQ 
of 110 on the WAIS-III, the 90% confidence level interval is between 96 and 113 
(Wechsler, 1997, p. 198). The availability of this information in test manuals en- 
courages test users to apply confidence intervals in interpreting scores without 
having to calculate them. However, the determination of whether or not the pub- 
lished figures are applicable and meaningful in each instance of test use still rests 
with the user. 

Table 4.1 and Figure 4.1 illustrate how the score reliability and SEM data might 
be used in the analysis of 4—out of a total of 14—-subtest scores one can obtain 
from the WAIS-IIL In addition to Maria’s Vocabulary and Information subtest 
scores, already encountered in the previous examples, two other fictitious WAIS- 
III subtest scores, namely, Arithmetic and Digit Span, have been added to her pro- 
file. These subtests were selected because the primary abilities they assess—quan- 
titative ability and short-term auditory memory, tespectively—are sufficiently 
unique to make a comparison among them and the other two scores both inter- 
esting and plausible. The error bands calculated at the 90% confidence level, 
based on the SEMs for the respective subtests, are presented in Table 4.1. The 
SEMs, in turn, have been calculated based on combinations of a//the pertinent re- 
liability figures presented in the WALS-I/[/WMS-I Technical Manual for each of 
the four subtests (Psychological Corporation, 1997). This more rigorous practice 
is in contradistinction to the use of the (smaller) SEMs, based on only one estimate 
of reliability estimate, which are provided in the tables of the test manual for IQ 
and index scores of the WAIS-III (see Wechsler, 1997, pp. 195-202). Figure 4.1 
displays the data from Table 4.1 in graphic form. Inspection of this figure quickly 
reveals that once the SEM is taken into account the probable ranges of Maria’s 
scores overlap considerably, which means that some of the score differences ob- 
tained may be due to measurement error. At the same time, this sort of profile 
analysis allows the test user to explore hypotheses about the possible meaning of the 
differences in Maria’s performance on the abilities tapped by the subtests whose 
error bands do not overlap (i.e., Vocabulary and Information, Arithmetic and 
Digit Span, and Digit Span and Information). It appears, for example, that Maria 
may have relative strengths in short-term memory capacity and vocabulary, 
whereas her store of general information may be a relative weakness. Naturally, 
any conclusions based on such differences are subject to confirmation or revision 
in light of additional data. Nevertheless, when a psychological assessment calls for 
an evaluation of an individual’s possible strengths and weaknesses either in the in- 
tellectual area or in some other tespect—vocational interests, for example—this 
type of exploratory analysis, while not definitive, can be quite helpful. 
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WAIS-III Subtests Subtest Scaled Scores 


Vocabulary 
Arithmetic 


Digit Span 


123 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 


= 


n 
= 


Information = 


Figure 4.1 Graphic profile of Maria’s WAIS-III subtest scores, illustrating the 


use of error bands in Table 4.1 


The standard error of the difference between scores (SE 


ueg» discussed earlier, 


serves a similar purpose as the profile analysis displayed in Figure 4.1. It provides 


data regarding score discrepancies that may have practical or psychological signif- 


icance. Typically, the SE, 


iff 


values—also found in many test manuals—are used to 


evaluate the statistical significance of the obtained differences between scores that 
are of special interest. The manuals of the recent editions of the Wechsler scales, 
for instance, routinely include tables showing the point differences in IQs (Verbal 
versus Performance IQs) and index scores needed for statistical significance at the 
90% and 95% confidence levels. Recognizing that a statistically significant differ- 


ence may not necessarily be psychologi- 
cally significant, the authors of these 
test manuals—and those of other 
tests as well—also provide tables 
showing the frequencies with which 
score differences of various magni- 
tudes were found among the stan- 
dardization samples of the tests in 
question. This information addresses 
the question of how common or how 
rate any given difference is among the 
normative sample. Its importance de- 
tives from the assumption (not always 
justified but worth pondering) that if 
differences of a certain magnitude oc- 
cut frequently, they probably have less 
interpretive significance than those 
that occur rarely. 


DON’T FORGET 


The use of standard errors of mea- 
surement (SEMs) for test scores, and 
standard errors for test score differ- 
ences (SE,,,.), both of which are de- 
rived from estimates of score reliabil- 
ity, are essential pieces of information 
because 


|. SEMs provide confidence intervals 
for obtained test scores that alert 
test users to the fact that scores 
are subject to fluctuation due to 
measurement error, and 


2. the confidence intervals obtained 
with the use of SE... statistics fore- 
stall the overvaluation of score dif- 
ferences that may be insignificant in 
ight of measurement error. 
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The SE 


diff 
tained difference between the scores of two individuals on the same test may be 


can, of course, also be used to calculate the probability that an ob- 


due to measurement error. For instance, in educational- or employment-selection 
decisions made with the help of tests, differences between the scores of candi- 


dates may be evaluated for significance in light of the SE, in addition to other 


diff? 
relevant factors. Once again, as with profile analysis, this process calls attention 


to the fact that obtained score differences cannot be taken at face value. 


The Relationship Between Reliability and Validity 


From the psychometric perspective, evidence of score reliability is considered to 
be a necessary, but not sufficient, condition for validity (see, e.g., Sawilowsky, 
2003). In fact, as we shall see in the next chapter, the two concepts are intrinsi- 
cally related in that score reliability can in itself be seen as minimal evidence that 
a valid measure of a sample of behavior has been attained. 

Assessment professionals generally agree that evidence of score reliability is not 
a sufficient basis on which to make valid inferences about the meaning of scores. 
However, some room for disagreement exists with regard to the extent to which 
reliability evidence is seen as essential for the valid appraisal of all the types behav- 
ior samples that can be gathered through tests. For instance, when test scores are 
derived from behavior samples that are unique or idiosyncratic, they may not be 
repeatable or consistent. Tests that call forth an individual’s optimal level of per- 
formance, such as work samples or portfolios, may produce results that are valid 
and reliable in terms of accuracy and precision but not in terms of consistency or 
stability (see, e.g., Moss, 1994). Similarly, instruments that are individually admin- 
istered, like many intelligence scales or projective techniques, are highly suscep- 
tible to influences stemming from the quality of rapport between the examiner and 
the test taker, as well as other motivational and situational factors. In the context 
of individual assessment, such instruments may provide valid glimpses into as- 
pects of a person’s psychological make-up that might not be reproduced with a dif- 
ferent examiner or in a different circumstance, even if standardized procedures are 


rigidly observed (Masling, 1960; McClelland, 1958; Smith, 1992). 


CONCLUSION 


The use of psychological tests would be greatly simplified if reliability coefficients 
and SEMs could be taken at face value in evaluating test scores. As this chapter at- 
tests, however, the reliability of scores is a relative judgment based both on psy- 
chometric data and on the context within which tests are administered. We shall 
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see in Chapter 5 that the same is true of the validity of test score data. Thus, al- 
though availability of appropriate psychometric data on score reliability is a basic 
prerequisite for any use of test scores, the context within which psychological 
testing takes place is also a fundamental consideration in the interpretation of ob- 
tained scores of individuals or groups. As the potential impact of decisions to be 
made with the help of test scores increases, both of these factors assume greater 
importance. 


el TEST YOURSELF AAA 


l. A true score is 
(a) a hypothetical entity. 
(b) areal entity. 
(c) equal to the observed score. 
(d) equal to the observed score plus error 


2. If the reliability of a test is well established, test users can assume that 
the scores obtained from that test will be reliable. True or False? 


3. Which of the following sources of error in test scores is not assessed by 
traditional estimates of reliability? 


a) Interscorer differences 
b) Time sampling 
c) Content sampling 


d) Deviations from standardized procedures 


4. reliability coefficients are used to estimate time sampling 
error in test scores. 


a) Test-retest 

b) Alternate form 

c) Scorer 

d) Split-half 

5. Which of the following types of reliability coefficients results in a com- 
bined estimate of error stemming from two different sources? 


a) Scorer 
b) Test-retest 
c) Alternate form 


d) Delayed alternate form 


(continued ) 
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. All other things being equal, scores obtained from longer tests are 


those obtained from comparable tests that are shorter. 
(a) less reliable than 
(b) more reliable than 
(c) just as reliable as 


. The magnitude of a reliability coefficient is more likely to be affected by 


the than by the of the sample for which it is com- 
puted. 


a) size/heterogeneity 
b) heterogeneity/size 


One of the distinct advantages of generalizability theory over traditional 
approaches to score reliability is that generalizability theory 


a) requires a smaller number of observations. 
b) results in smaller error components. 


c) allows for the evaluation of interaction effects. 


d) uses less complicated statistical methods. 


. Suppose that a student obtains a score of 110 on a test with M = 100, SD 


= 20, and an estimated reliability of .96. Chances are 68 out of 100 that 
the student’s true score falls somewhere between 


(a) 100 and 110. 
(b) 102 and 112. 
(©) 106 and 114. 
(d) 110 and 120. 


. The standard error of measurement of Test A is 5 and the standard er- 


ror of measurement of Test B is 8. The standard error of the difference 
for comparing scores from the two tests will be 


(a) less than 8. 

(b) less than 5. 

(c) between 5 and 8. 
(d) greater than 8. 


Answers: |. a; 2.b; 3. d;4.a; 5. d; 6. b; 7.b; 8.¢;9.c 10.d. 


Five 


ESSENTIALS OF VALIDITY 


P 


bear to support any inference that is to be made on the basis of test results. The 


prima 


the placement of this topic in the first chapter, which defines validity as “the de- 
gree to which all the accumulated evidence supports the intended interpretation 
of test scores for the proposed purpose” (AERA, APA, NCME, 1999, p. 11). Im- 
plicit in this definition are three interrelated ideas that reflect the testing profes- 


sion’s 


1. 


3. 


I5| 


sychological tests exist in order to help us draw inferences about people and 
their behavior. Validity—which is, by far, the most fundamental issue re- 
garding test scores and their uses—hinges on the evidence we can bring to 


cy of validity considerations is recognized in the current Testing Standards by 


current views about this central and multifaceted concept: 


The validity of test scores stems from all the accumulated evidence to 
support their interpretation and uses. Thus, validity is always a matter 
of degree rather than an all-or-none determination. Va/idation—which 
is the process whereby validity evidence is gathered—begins with an 
explicit statement by the test developer of the conceptual framework 
and rationale for a test, but is in its very nature open-ended because it 
includes all the information that adds to our understanding of test re- 
sults. 


. As the theoretical understanding of and empirical evidence for test 


score interpretations accumulates, the validity of inferences (i.e., hy- 
potheses) made on the bases of test scores for various proposed pur- 
poses may be enhanced or diminished. A corollary to this notion, ex- 
plicitly stated in the Testing Standards (AERA, APA, NCME, 1999), is 
that “validation is the joint responsibility of the test developer [who 
provides evidence and a rationale for the intended use of the test] and 
the test user [who evaluates the available evidence within the context 
in which the test is to be used]” (p. 11). 

Because of the many different purposes to which test scores may be 
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DON’T FORGET 


Perhaps no other theorist has been more influential in reshaping the concept of 
validity than Samuel Messick. According to Messick (1989, p. 13), “validity is an in- 
tegrated evaluative judgment of the degree to which empirical evidence and the- 
oretical rationales support the adequacy and appropriateness of inferences and ac- 
tions based on test scores or other modes of assessment.” 


applied, the evidentiary bases for test score interpretations can be de- 
rived through a variety of methods. Contributions to the validity evi- 
dence of test scores can be made by any systematic research that sup- 
ports or adds to their meaning, regardless of who conducts it or when 
it occurs. As long as sound scientific evidence for a proposed use of 
test scores exists, qualified test users are free to employ scores for their 
purposes, regardless of whether these were foreseen by the developers 
of the test. This proposition helps to explain the multifaceted nature 
of validation research, as well as its often redundant and sometimes 
conflicting findings. It also accounts for the longevity of some instru- 
ments, such as the MMPI and the Wechsler scales, for which a vast 
literature—encompassing numerous applications in a variety of 
contexts—has been accumulated over decades of basic and applied 
research. 


The alert reader may have gathered at this point that validity, just as reliability, 
is not a quality that characterizes tests in the abstract or any specific test or test 
data. Rather, validity is a matter of judgments that pertain to test scores as they are 
employed for a given purpose and in a given context. Hence, the process of vali- 
dation is akin to hypothesis testing: It subsumes the notions of test score mean- 
ing, and test score reliability, discussed in the two previous chapters, as well as the 
ways in which the applications of test data to psychological research and practice 
can be justified, the topic covered in the present chapter. Rapid Reference 5.1 lists 
some of the most significant contributions to the topic of validity from the 1950s 
to the 1990s. 


HISTORICAL PERSPECTIVES ON VALIDITY 


The rise of modern psychological testing took place at about the same time that 
psychology was becoming an established scientific discipline. Both fields date 
their beginnings to the late part of the 19th and early years of the 20th centuries. 
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Basic References on Validity 


Samuel Messick articulated his views on validity most explicitly in a chapter that 
appeared in Educational Measurement (3rd ed., pp. 13—103), a notable volume 
edited by Robert L. Linn and published jointly by the American Council on Educa- 
tion and Macmillan in 1989. Messick’s Validity chapter, and his other works on the 
topic (e.g, Messick, 1988, 1995), have directly influenced its treatment in the cur- 
rent version of the Testing Standards (AERA, APA, NCME, 1999). Other key con- 
tributions that are widely acknowledged as having shaped the evolution of theo- 
retical concepts of validity include the following: 


e Cronbach, L. J., & Meehl, P E. (1955). Construct validity in psychological tests. 
Psychological Bulletin, 52, 28|—302. 

e Loevinger, J. (1957). Objective tests as instruments of psychological theory 
[Monograph Supplement]. Psychological Reports, 3, 635—694. 


e Embretson, S. (1983). Construct validity: Construct representation versus 
nomothetic span. Psychological Bulletin, 93, 179-197. 


e Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & 
H. I. Braun (Eds.), Test validity (pp. 3-17). Hillsdale, NJ: Erlbaum. 


As a result of this historical coincidence, our understanding of the nature, func- 
tions, and methodology of psychological tests and measurements has evolved 
over the past century in tandem with the development and growing sophistica- 
tion of psychological science. 

At the beginning, scientific psychology was primarily concerned with estab- 
lishing psychophysical laws, through the experimental investigation of the func- 
tional relationship between physical stimuli and the sensory and perceptual re- 
sponses they arouse in humans. Theoretical psychology consisted primarily of 
armchair speculation of a philosophical nature, until well into the first quarter of 
the 20th century. Neither of these statements implies that the contributions of the 
pioneers of psychology were not of value (see, e.g., Boring, 1950; James, 1890). 
Nevertheless, arising against this backdrop, the first psychological tests came to 
be seen, somewhat naively, as scientific tools that measured an ever-expanding 
catalog of mental abilities and personality traits in much the same way as the psy- 
chophysicists were measuring auditory, visual, and other sensory and perceptual 
responses to stimuli such as sounds, light, and colors of various types and inten- 
sities. Furthermore, as we saw in Chapter 1, the success that the Stanford-Binet 
and the Army Alpha had in helping to make practical decisions about individuals 
in education and employment settings led to a rapid proliferation of tests in the 
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first two decades of the 20th century. The wide range of applications for which 
these instruments were used soon overtook the theoretical and scientific ratio- 
nales for them that were available at the time. In short, many early psychological 
tests were developed and used without the benefit of the psychometric theory, 
ethical principles, and practical guidelines that would begin to accumulate in later 
decades (von Mayrhauser, 1992). 


The Classic Definition of Validity 


Recognition of this state of affairs within the profession resulted in the first at- 
tempts to delineate the characteristics that would distinguish a good test from a 
bad one. Thus, the first definition of validity as “the extent to which a test measures 
what it purports to measure” was formulated in 1921 by the National Association 
of the Directors of Educational Research (T. B. Rogers, 1995, p. 25). It was ratified 
by many testing experts—including Anne Anastasi in all the editions of her influ- 
ential textbook on Psychological Testing (1954—1988) as well as Anastasi and Urbina 
(1997, p. 8). The view that “test validity concerns what the test measures and how 
wellit does so” (Anastasi & Urbina, p. 113) is still regarded by many as the heart of 
the issue of validity. In spite of its apparent simplicity, this view poses a number of 
problems, especially when it is seen from the perspective of the current Testing Stan- 
dards (AERA, APA, NCME, 1999) and of the flux that still exists with regard to 
defining some of the most basic constructs within the field of psychology. 


Problematic Aspects of the Traditional View of Validity 
The issues that the classic definition of validity raises revolve around its unstated 
but clear assumptions that 


1. validity is a property of tests, rather than of test score interpretations; 

2. in order to be valid, tests scores should measure some purported con- 
struct directly; and 

3. score validity is, at least to some extent, a function of the test author’s 
ot developer’s understanding of whatever construct she or he intends 
to measute. 


While these assumptions may be justified in certain cases, they definitely are 
not justified in every case. The first assumption, for instance, is tenable only as 
long as validation data support the stated purpose of the test and as long as the 
test is used specifically for that purpose and with the kinds of populations for 
which validity data have been gathered. The second and third assumptions are 
justified only for tests that measure behavior which can be linked to psychologi- 
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cal constructs in fairly unequivocal ways, such as certain memory functions, 
speed and accuracy in the performance of various cognitive processing tasks, or 
extent of knowledge of a well-defined content universe. They are not necessarily 
tenable for (a) tests designed to assess multidimensional or complex theoretical 
constructs about which there is still much debate, such as intelligence or self- 
concept; (b) tests developed on the basis of strictly empitical—as opposed to 
theoretical or logical—relationships between scores and external criteria, such as 
the original MMPI; or (c) techniques whose purpose is to reveal covert or un- 
conscious aspects of personality, such as projective devices. For instruments of 
this nature, what is being measured is behavior that can be linked more ot less di- 
rectly to the constructs that are of real interest, primarily through a network of 
correlational evidence. Rapid Reference 5.2 defines the various meanings of the 
word construct and may help to clarify the distinctions just made, as well as those 
to come later in this chapter. 

The idea that test score validity is a function of the degree to which tests mea- 
sure what they purport to measure also leads to some confusion between the con- 
sistency or precision of measurements (i.e., their reliability) and their validity. As 
we saw in Chapter 4, if a test measures whatever it measures wel, its scores may 
be deemed to be reliable (consistent, precise, or trustworthy), but they are not 
necessarily valid in the contemporary, fuller sense of the term. In other words, test 
scores may be relatively free of measurement error, and yet may not be very use- 
ful as bases for making the inferences we need to make. 

Moreover, the implication that a test score reflects what the test author intends 
it to reflect has been as a source of additional misunderstandings. One of them 
concerns the titles of tests, which should never be—but often are—taken at face 
value. Test titles range from those that ate quite accurate and empirically defen- 
sible to those that merely reflect test authors’ (unfulfilled) intentions or test pub- 
lishers’ marketing concerns. A second, and even more important, problem with 
the notion that valid test scores reflect their expressed purpose is that it can lead 
to superficial or glib empirical definitions of psychological constructs. Possibly 
the most famous example of this is E. G. Boring’s 1923 definition of intelligence as 
“whatever it is that intelligence tests measure” (cited by Sternberg, 1986, p. 2). 

As a result of these misunderstandings, the field of psychological testing has 
been saddled with instruments—purporting to measure ill-defined or faddish 
constructs—whose promises vastly overstate what they can deliver, whose use in 
psychological research impedes or delays progress in the discipline, and whose 
existence, by association, diminishes the image of the field as a whole. Early mea- 
sures of masculinity-femininity are a prime example of this sort of problem (Con- 
stantinople, 1973; Lenney, 1991; Spence, 1993), although there are many others. 
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Deconstructing Constructs 


Because the term construct is used so frequently in this chapter, a clarification of 
its meaning is necessary at this point. Generally speaking, a construct is anything 
that is devised by the human mind but not directly observable. Constructs are ab- 
stractions that may refer to concepts, ideas, theoretical entities, hypotheses, or in- 
ventions of many sorts. 


In psychology, the word construct is applied to concepts, such as traits, and to the 
theoretical relationships among concepts that are inferred from consistent em- 
pirical observations of behavioral data. Psychological constructs differ widely in 
terms of 


e their breadth and complexity, 


e their potential applicability, and 


e the degree of abstraction required to infer them from the available data. 


As a rule, narrowly defined constructs require less abstraction but have a smaller 
range of application. Moreover, since it is easier to obtain consensual agreement 
about constructs that are narrow, simple, and less abstract, these are also more 
easily assessed than broader and multifaceted constructs that may have acquired 
different meanings across diverse contexts, cultures, and historical periods. 


Examples: 


e Whereas manual dexterity is a construct that can be linked to specific behav- 
ioral data readily, creativity is far more abstract. Thus, when it comes to evaluat- 
ing these traits, determining who has greater manual dexterity is much easier 
than determining who is more creative. 


* Introversion is a simpler and more narrowly defined construct than conscien- 
tiousness. While the latter is potentially useful in predicting a broader range of 
behaviors, it is also more difficult to assess. 


Synonyms: The terms construct and latent variable are often used interchangeably. 
A latent variable is a characteristic that presumably underlies some observed phe- 
nomenon but is not directly measurable or observable. All psychological traits are 
latent variables, or constructs, as are the labels given to factors that emerge from 
factor analytic research, such as verbal comprehension or neuroticism. 


Perhaps the most significant consequence of the traditional definition of va- 
lidity is that it became attached to tests and to what they purport to measure, 
rather than to test scores and the interpretations that could justifiably be based on 
them. By implication, then, any evidence labeled as “est validity came to be seen as 
proof that the test was valid and worthy of use, regardless of the nature of the link 
between test score data and the inferences that were to be drawn from them. Con- 
sequently, innumerable studies in the psychological literature have used scores 
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from a single instrument to classify research participants into experimental 
groups, many clinicians have relied exclusively on test scores for diagnosis and 
treatment planning, and an untold number of decisions in educational and em- 
ployment settings have been based on cutoff scores from a single test. Too often, 
choices like these are made without considering their appropriateness in specific 
contexts or without reference to additional sources of data and justified simply 
on the basis that the test in question is supposed to be “a valid measure of... 2” 
whatever its manual states. 

An important signpost in the evolution of the concept of validity was the pub- 
lication of the Technical Recommendations for Psychological Tests and Diagnostic Techniques 
(APA, 1954), the first in the series of testing standards that were retitled, revised, 
and updated in 1955, 1966, 1974, 1985, and most recently, in 1999 (AERA, APA, 
NCME). With each subsequent revision, the Testing Standards—pteviously dis- 
cussed in Chapter 1—have attempted to promote sound practices for the con- 
struction and use of tests and to clarify the basis for evaluating the quality of tests 
and testing practices. 

The Technical Recommendations published in 1954 introduced a classification of 
validity into four categories to be discussed later in this chapter: content, predic- 
tive, concurrent, and construct validity. Subsequently, the 1974 Standards reduced 
these categories to three, by subsuming predictive and concurrent validity un- 
der the rubric of criterion-related validity, and further specified that content, 
criterion-related, and construct validity are aspects of, as opposed to Apes of, valid- 
ity. In the same year, the Standards also introduced the notion that validity “refers 
to the appropriateness of inferences from test scores or other forms of assess- 
ment” (APA, AERA, NCME, 1974, p. 25). 

In spite of the specifications proposed by the 1974 Standards more than a 
quarter century ago, the division of validity into three types (which came to be 
known as the ¢ripartite view of validity) became entrenched. It has survived up to 
the present in many test manuals and test reviews, as well as in much of the re- 
search that is conducted on psychometric instruments. Nevertheless, successive 
revisions of the Standards—especially the current one—have added stipulations 
that make it increasingly clear that whichever classification is used for validity 
concepts should be attached to the types of evidence that are adduced for test 
score interpretation rather than to the tests themselves. With this in mind, we 
turn now to a consideration of the prevailing view of validity as a unitary con- 
cept and to the various sources of evidence that may be used to evaluate possible 
interpretations of test scores for specific purposes. For further information on 
the evolution of validity and related concepts, see Anastasi (1986), Angoff 
(1988), and Landy (1986). 
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CURRENT PERSPECTIVES ON VALIDITY 


Beginning in the 1970s and continuing up to the present, there has been a con- 
certed effort within the testing profession to refine and revise the notion of va- 
lidity and to provide a unifying theory that encompasses the many strands of 
evidence from which test scores derive their significance and meaning. One 
consistent theme of this effort has been the integration of almost all forms of va- 
lidity evidence as aspects of construct validity (Guion, 1991; Messick, 1980, 1988, 
1989; Tenopyr, 1986). This, in turn, has prompted a reexamination of the mean- 
ing of construct—defined in general terms in Rapid Reference 5.2—as it applies 
specifically in the context of validity in psychological testing and assessment (see, 
e.g., Braun, Jackson, & Wiley, 2002; Embretson, 1983). 


The Integrative Function of Constructs in Test Validation 


In psychological testing, the term construct has been used, often indistinctly, in two 
alternate ways: 


1. To designate the /raits, processes, knowledge stores, or characteristics whose 
presence and extent we wish to ascertain through the specific behavior 
samples collected by the tests. In this meaning of the word, a construct 
is simply what the test author sets out to measure—that is, any hypo- 
thetical entity derived from psychological theory, research, or observa- 
tion of behavior, such as anxiety, assertiveness, logical reasoning ability, 
flexibility, and so forth. 

2. To designate the żnferences that may be made on the basis of test scores. 
When used in this way, the term construct refers to a specific interpreta- 
tion of test data, or any other behavioral data—such as the presence 
of clinical depression or a high probability of success in some en- 
deavor—that may be made based on a network of preestablished the- 
oretical and empirical relationships between test scores and other vari- 
ables. 


Several theorists have tried to explain how these two meanings relate to the 
notion of test score validity. One of the earliest formulations was Cronbach’s 
(1949) classification of validity into two types, namely, logical and empirical. Sub- 
sequently, in an influential paper he coauthored with Meehl in 1955, Cronbach 
suggested the use of the term construct validity to designate the nomological net, or 
network of interrelationships between and among theoretical and observable el- 
ements that support a construct. In an attempt to clarify how these two meanings 
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could be distinguished in the process of test development, construction, and eval- 
uation, Embretson (1983) proposed a separation between two aspects of con- 
struct validation research, namely, construct representation and nomothetic 
span. According to Embretson (p. 180), construct representation research “is con- 
cerned with identifying the theoretical mechanisms that underlie task perfor- 
mance.” From an information-processing perspective, the goal of construct rep- 
resentation is task decomposition. The process of task decomposition can be applied 
to a variety of cognitive tasks, including interpersonal inferences and social judg- 
ments. It entails an examination of test responses from the point of view of the 
processes, strategies, and knowledge stores involved in their performance. Nomo- 
thetic span, on the other hand, concerns “the network of relationships of a test to 
other measures” (Embretson, p. 180); it refers to the strength, frequency, and 
pattern of significant relations between test scores and other measures of the 
same—ot different—traits, between test scores and criterion measures, and so 
forth. 

Embretson (1983) described additional features of the concepts of construct 
representation and nomothetic span that help to clarify the differences between 
these two aspects of construct validation research. Two of the points she made, 
concerning the distinction between the functions the two kinds of research can 
serve, are particularly useful in considering the role of sources of validity evi- 
dence: 


1. Construct representation research is concerned primarily with identifying differences 
in the test’s tasks, whereas nomothetic span research is concerned with differences 
among test takers. In consttuct representation research, a process, strat- 
egy, ot knowledge store identified through task decomposition (e.g,, 
phonetic coding, sequential reasoning, or ability to comprehend 
elementary-level texts) may be deemed essential to the performance of 
a test task, but yield no systematic differences across a test-taking pop- 
ulation made up of readers. On the other hand, in order to investigate 
the nomothetic span of test scores (i.e., the network of relationships 
between them and other measures), it is necessary to have data on indi- 
vidual differences and variability across test takers. This reinforces the 
crucial importance that score variability has for deriving information 
that can be used to make determinations or decisions about people, 
which was discussed previously in Chapters 2 and 3. If the scores of a 
group of people on a test designed to assess the ability to comprehend 
elementary level texts, for instance, are to be correlated with anything 
else—or used to determine anything other than whether these people, 
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as a gtoup, possess that ability—there must be some variability in the 
scores. 

2. Validation of the construct-representation aspect of test tasks is independent of the 
supporting evidence that may be gathered in terms of the nomothetic span of test 
scores, and vice versa. In other words, while we may know precisely what 
processes are involved in the performance of test tasks, absent signifi- 
cant correlations with meaningful extratest behaviors or measures, test 
scores may be of limited use. By the same token, it is possible to obtain 
a strong network of relationships between test scores and other mea- 
sures, without having a clear notion of the construct that those scores 
represent. The example Embretson (1983) uses is that of intelligence 
test scores, which have a strong nomothetic span (in that they consis- 
tently correlate more or less strongly with a variety of other measures) 
but still have relatively unclear theoretical bases. 


The conceptual scheme outlined by Embretson (1983) retains the notion of 
construct validation as a unitary and comprehensive way of expressing the scien- 


DON’T FORGET 


People make inferences based on observations and behavior samples all the time. 

For example, if we hear someone speak using poor grammar we may infer that 

the person is uneducated. If a person is invariably on time for appointments, we 

may infer that she or he is punctual. Some of our inferences are correct and some 

are not. Some matter, and some do not. 

If the inferences we make matter enough for us to wish to ascertain their correct- 

ness, and thereby validate them, we need to 

|. define our terms unequivocally (e.g, What do we mean by “‘uneducated’”? 
Does “being on time invariably” capture the entire concept of punctuality’); 

2. investigate the reliability of our observations (e.g., Does the person always use 


poor grammar or only in some circumstances? Is our friend on time for all ap- 
pointments or only for those we have had a chance to observe?); and 


3. decide whether there is sufficient evidence to justify the inferences we wish to 
make, based on our definitions and on the available data (e.g, Being on time 
for all appointments is a sufficient basis on which to judge a person’s punctual- 
ity) or whether we need to corroborate our inference with additional data 
(e.g, Does the person display other indexes of what we mean by “‘unedu- 
cated’”?). 

Psychological tests are tools designed to help refine and quantify behavioral ob- 

servations for the purpose of drawing inferences about individuals, groups, or psy- 

chological constructs. Fundamentally, psychological test scores are valid to the ex- 
tent that they can help us draw accurate inferences. 


< ESSENTIALS OF VALIDITY 161 


tific approach to the integration of any evidence bearing on the meaning or in- 
terpretation of test scores. At the same time, it provides a basis for distinguishing 
between (a) the sources of evidence for test score validity that bear primarily on 
knowing what we are measuring (i.e., construct representation) and (b) those that 
deal mainly with the inferences we can make based on what we are measuring (Le., 
nomothetic span). It should be noted that these sources of evidence may be, and 
often are, interrelated and that both involve theoretical and observable elements, 
as well as models or postulates concerning the interrelationships among ele- 
ments. 


SOURCES OF VALIDITY EVIDENCE 


In general, the essence of judgments about the validity of test scores centers on 
the relationship between what the scores represent and the questions test users 
need to answer with the use of tests. The questions we pose determine the type 
of evidence we need as well as the logical relationships—inductive and deduc- 
tive—that have to be established to address the issues of (a) what we are measur- 
ing with tests and (b) what inferences we can draw from test scores. It should also 
be clear at this point that the greater the significance or potential impact of the 
answers we want, the more convincing the evidence needs to be. In the remain- 
der of this chapter, we deal with the types of evidence required for validating test 
score inferences, with the understanding that the proposed interpretation of a 
test score determines the conceptual framework for its validation. Table 5.1 pre- 
sents a list of the major categories under which aspects of validity may be classi- 
fied, along with the principal sources of evidence for each, which are discussed in 
the remainder of this chapter. It is important to recognize at the outset that nei- 
ther the aspects of validity nor the sources or types of evidence associated with 
them are mutually exclusive. Validation strategies should, in fact, incorporate as 
many sources of evidence as practicable or as appropriate to the purposes of a 
test. 


Validity Evidence Based on Test Content and Response Processes 


Some psychological tests are designed to sample behavior that can be linked more 
ot less directly to the inferences we wish to make based on their scores. By and 
large, these instruments fall in the category of content- or criterion-referenced 
tests, already discussed at some length in Chapter 3. As noted there, most of these 
tests are used in educational and occupational settings, although they can also be 
applied in fields (e.g., neuropsychological assessment) where it is necessary to as- 
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Table 5.1 Aspects of Construct Validity and Related Sources of Evidence 


Aspect of 
Construct Validity Sources of Evidence’ 
Content-related Relevance and representativeness of test content and task re- 


sponse processes 
Face validity (i.e., superficial appearances) 


Patterns of convergence Internal consistency of test results and other measures of re- 
and divergence liability 


Correlations among tests and subtests 
Multitrait-multimethod matrix 


Score differentiation consonant with expected differences 
based on age or other status variables 


Experimental results (i.e., correspondence between test 
scores and the predicted effects of experimental interven- 
tions or theory-based hypotheses) 
Exploratory factor analysis 
Structural equation modeling techniques 

Criterion-related Accuracy of decisions based on concurrent validation (i.e., 
correlations between test scores and existing criteria) 
Accuracy of decisions or predictions based on predictive val- 


idation (i.e., correlations between test scores and predicted 
criteria) 


“See Chapter 5 text for explanations of terms 


certain whether a person is able or unable to perform tasks that may hold diag- 
nostic significance. These tests are made up of items that either sample knowl- 
edge from a defined content domain or require test takers to demonstrate that 
they possess a given ability or competence in some skill. Validation procedures 
for tests of this type are in many ways the simplest and most agreed-upon aspect 
of test development because the evidence on which inferences ate to be made can 
be defended on logical grounds as well as by demonstrable relationships between 
the content of the test and the construct that the test is designed to represent. 

Evidence of test score validity that derives from the content of the test can be 
built into a new instrument at the outset, by the choice of items or tasks that are 
included in the test. The primary requirement for developing tests of this sort is 
a careful specification of the content domains, cognitive processes, skills, or types 
of performance to be sampled by the test and of their relative importance or 
weight. 
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° In the educational context, examples of such specifications can be 
found in school curricula, course syllabi, textbooks, and any other materi- 
als that outline, define, or prioritize the outcome objectives of educational 
or training experiences in terms of both content knowledge and perfor- 
mance capabilities. The process of delimiting knowledge domains and de- 
termining the desired outcomes of instruction is within the purview of 
teachers, trainers, and other subject matter experts who determine curric- 
ula or write the textbooks in various disciplines. 

e In occupational settings, the specifications of the skill or knowledge 
domains to be sampled by the test are based on job analyses. Job analysis 
refers to any one of a variety of methods aimed at discovering the nature of 
a given job through the description of the elements, activities, tasks, and du- 
ties related to it (see, e.g., Brannick & Levine, 2002). Job analysis method- 
ology has a variety of applications in the management of human resources, 
including performance appraisal and the determination of training needs 
among others. Within the context of occupational selection and classifica- 
tion, job analyses—based on input from employers, job supervisors, and/ 
or current employees—ate used to delineate the abilities, skills, and knowl- 
edge stores required for job performance. 

e In neutopsychological assessment, the specifications of the processes 
and cognitive capabilities to be assessed are derived from theoretical and 
empirical knowledge of the linkages between the central nervous system 
and behavioral functions. The nature of the content of neuropsychological 
assessment tools is based on accumulated scientific and clinical evidence 
about brain-behavior relationships. 


Rapid Reference 3.7, in Chapter 3, lists some simple examples of objectives 
and items typical of domain-referenced tests. More extensive examples, including 
tables of specifications for content-referenced tests and guidance in how they are 
ptepared, are available in Gronlund (2003) and Linn and Gronlund (1995, 
pp. 119-125). Regardless of the setting in which a content-referenced test is ap- 
plied, once the specifications for the knowledge, skills, or processes to be gauged 
through the test have been set, content validation procedures involve the critical 
review and examination of test content from two perspectives. The first is the re/- 
evance of the content sampled by the test to the specified domain; the second one 
is the representativeness of the content sampled by the test with regard to the spec- 
ifications about the domain that it is designed to cover. Although the issue of rel- 
evance must tely on a consensus of subject matter experts, it can also be sup- 
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ported by empirical findings, such as differences in the scores of students in suc- 
cessive grades or individuals at various stages of a training process. The authors 
and developers of tests must support claims of the content-related validity of test 
scores in manuals, technical handbooks, and other such sources of supporting 
documentation for tests. When the primary basis of an instrument’s validation 
evidence centers on the specific content, skills, or cognitive processes it assesses, 
a description of the systematic procedures used to ensure the relevance and rep- 
resentativeness of test content to the specified domains is required. Rapid Refer- 
ence 4.3 presents an example of the role that inadequate representativeness of 
content coverage can play in undermining both the reliability and validity of 
content-referenced test scores. 


Educational Testing 

Scores that derive their validity from a direct and demonstrable connection be- 
tween test content and the specifications used in developing the test abound at all 
levels of education and training in which the outcomes of instruction can be un- 
ambiguously defined. Almost all teachet-designed classroom tests fit into this cat- 
egory, as do many of the standardized tests published by ETS, the ACT, and simi- 
lar organizations. The primary purpose of these instruments is to gauge educational 
achievement—that is, what students have learned through their schooling. Scores 
from these tests can answer most directly questions such as “How much of the 
specified domain has the learner mastered?” or “What degree of mastery or profi- 
ciency has the test taker attained in the skill in question?’ Content- or domain- 
referenced instruments may be applied for a variety of decisions, including assign- 
ing grades in a course, providing course credits through examination, conferring a 
degree or diploma after a program of study, certifying or licensing individuals to 
practice in a given field, or even ascertaining readiness to undertake a more ad- 
vanced level of training, Typically, the decisions based on these tests hinge on the 
levels of mastery displayed by test takers. These levels of mastery, in turn, may be 
gauged in terms of percentage grades, percentile ranks in comparison to appropri- 
ate normative groups, or simple pass-fail determinations based on preestablished 
criteria, as discussed in Chapter 3. Rapid Reference 5.3 lists some typical examples 
of standardized educational tests, along with the purposes for which they were de- 
veloped (i.e., the knowledge and skills they are designed to evaluate) and their pri- 
mary applications. Sample questions and more elaborate descriptions of those tests 
are available on the Internet sites listed in Rapid Reference 5.3. 


Occupational Testing 
Many instruments used to select or place job applicants consist of job samples or 
simulations that actually call for performing the tasks of which the job is com- 
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posed (e.g., typing tests) or sample behaviors that can be directly linked to job 
performance by means of job analyses. Some of these tests are developed in- 
house by employers themselves and use local norms or performance criteria. 
Others are standardized instruments that provide normative scores for individu- 
als in various occupations and measure constructs of varying degrees of breadth. 
Numerous examples of these tests can be found in the Vocations section of the 
Classified Subject Index of Tests in Print VI (Murphy et al., 2002). Two that may 
be used to illustrate the diversity among tests of this type are described in Rapid 
Reference 5.4. 

Partly due to the cost and difficulty involved in developing and validating skill 
assessment instruments at the local level, the ACT (formerly American College 
Testing Program) organization (www.act.org) has initiated a program known as 
the WorkKeys system that combines a number of components aimed at helping 
businesses recruit, select, hire, and train employees. The job-profiling component 
allows job incumbents or their supervisors, in consultation with ACT experts, to 
select the most important tasks for a given job and identify the skills and levels of 
skills necessary for successful performance of that job. The assessment aspect of 
WorkKeys provides standardized instruments to evaluate applicants’ or employ- 
ees’ levels of skills in several critical areas, such as Applied Technology, Business 
Writing, Locating Information, Listening, and Teamwork, among others. The 
assessment of skills in Locating Information, for example, provides questions at 
four increasingly higher levels of complexity and is designed to measure skills 
ranging from finding information embedded in elementary workplace graph- 
ics—such as simple order forms, bar graphs, and floor plans—to drawing con- 
clusions from information presented in very detailed charts, tables, blueprints, 
and so forth. Based on comparisons of the information provided through these 
skill assessment tools and the minimum skill levels required by the job profiles, 
employers can evaluate the qualifications of applicants or the training needs of 
current employees. 


Evidence of Content Validity in Other Assessment Contexts 

The extent to which test items are relevant to and representative of a construct 
can be an additional source of validity evidence for instruments in almost any 
field. For instance: 


© In neuropsychological assessment, as mentioned earlier, specifications of the 
cognitive processes and behavior capabilities to be assessed are derived 
from well established theoretical and empirical knowledge of the rela- 
tionships between cognitive or behavioral functions and the neurologi- 
cal foundations that presumably underlie those functions. Thus, to a 
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large extent, the process of neuropsychological assessment relies on the 
experts’ knowledge of accumulated scientific evidence about brain- 
behavior relationships. A proper neuropsychological battery must in- 
clude items that tap a range of behaviors that is sufficiently broad and 
representative as to elicit evidence of functional capability, or impair- 
ment, in the various systems it is designed to assess (see, e.g., Franzen, 
2000; Lezak, 1995). The Boston Diagnostic Aphasia Examination 
(Goodglass, Kaplan, & Barresi, 2001), for example, provides a system- 
atic sampling of several communication functions, such as auditory 
comprehension and oral expression, in order to help in the diagnosis of 
aphasic syndromes and disorders of language. 

© In personality assessment, many self-report tools—such as checklists, in- 
ventories, and attitude or opinion surveys—rely to a large extent on the 
content of their items to help in generating hypotheses or drawing in- 
ferences about the particular constructs they aim to evaluate. Struc- 
tured observation procedutes, as well as various inventories used to col- 
lect data based on the reports of peers, parents, spouses, teachers, or 
other observers, also use item content as a basic source of validity evi- 
dence. Similarly, psychological tests designed to aid in the diagnosis of 
psychiatric disorders often include, or may even be entirely composed 
of, items that reflect critical symptomatic aspects of the syndromes they 
are designed to diagnose. Here again, the relevance and representative- 
ness of the items sampled by these instruments is of crucial importance 
in determining their usefulness for diagnostic purposes. Examples of 
tests of this type include the Beck Depression Inventory (BDI), the 
State-Trait Anxiety Inventory (STAD), the Symptom Checklist-90- 
Revised (SCL-90-R), and the Attitudes Toward Women Scale (AWS; 
Spence & Helmreich, 1972). 


Evidence of Validity From the Viewpoint of Test Takers 

The relevance and representativeness of test content is also pertinent with regard 
to an issue that is less substantive than score validity, but is nevertheless quite im- 
portant. Face validity refers to the superficial appearance of what a test measures 
from the perspective of a test taker or any other naive observer. All of the instru- 
ments discussed up to this point would have some face validity when used in the 
contexts we have been discussing. They would appear to test takers to be conso- 
nant with the stated educational, occupational, clinical, or investigative purposes 
of the assessment situations in which they are typically applied. Even though face 
validity is not necessarily an indication of validity from the psychometric per- 
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spective, it is nevertheless a desirable feature of tests because it promotes rapport 
and acceptance of testing and test results on the part of test takers. If the content 
of a test appears to be inappropriate or irrelevant to test takers, their willingness 
to cooperate with the testing process is likely to be undermined. Thus, test de- 
velopers need to consider the appearance of validity from the perspective of all 
parties—including test takers and other nonprofessionals—and, whenever 
possible, incorporate test content that seems relevant and appropriate to the 
situations in which a test will be used. 


Validity Evidence Based on Exploring Patterns 
of Convergence and Divergence 


As test score interpretation moves beyond direct and fairly unequivocal relation- 
ships between test content and the knowledge stores, skills, and functional pro- 
cesses the tests are designed to evaluate, it begins to rely on increasingly indirect 
sources of validity evidence. This is especially true for tests in the area of person- 
ality, not only because the constructs they assess are usually more theoretical and 
abstract than those assessed by cognitive tests, but also because test takers’ re- 
sponses to personality assessment tools are influenced by many more stylistic and 
situational determinants than cognitive test responses. 

There is a large and constantly expanding number of methods that can be used 
to augment the meaning of test scores beyond the relevance and representative- 
ness of test content. The common denominator of all of these procedures is that 
they produce evidence in the form of patterns of convergence and divergence be- 
tween test scores and other variables (see Table 5.1). Although a detailed expla- 
nation of these methods is well beyond the scope of this volume, a basic descrip- 
tion of the most frequently encountered procedures is in order. 


Test Score Reliability As a Source of Validity Evidence 

Investigations of the reliability of test scores from the point of view of stability, 
interscorer differences, content sampling error, and content heterogeneity may 
provide evidence concerning the cohesiveness, or distinctiveness, of test content. 
As discussed in Chapter 4, score reliability can in itself be seen as preliminary ev- 
idence that a trustworthy measure of a sample of behavior has been attained and 
thus can contribute indirect evidence of test score validity. If, for example, a test 
is designed to assess a unidimensional construct such as spelling ability, high in- 
ternal consistency coefficients would support the contention of unidimensional- 
ity. Similarly, if score consistency across different scorers can be achieved, one 
may suppose that they are all employing the same criteria and, thus, probably eval- 
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uating the same characteristics. If the construct being assessed is supposed to be 
stable—for instance, a personality trait or type—high test-retest score reliability 
would be an essential prerequisite for evidence of validity. 


Correlations Among Tests and Subtests 

A simple and frequently used way of gathering evidence that a particular test mea- 
sures the construct it purports to measure is by establishing high correlations be- 
tween its scores and those of other instruments that are meant to assess the same 
construct. One of the most basic examples of this type of procedure occurs when 
tests are revised and renormed. In such cases, test manuals almost invariably cite 
high correlations between the new and previous editions as evidence that both 
are measuring the same constructs. This is somewhat akin to computing correla- 
tions between alternate forms of a test to establish the reliability or consistency 
of scores across different forms of a test. It may be recalled from Chapter 3, how- 
ever, that even if correlations between the old and the revised versions of tests are 
very high, the normative scores for restandardized versions tend to drift in one 
direction or another due to changes in the population at different time periods. 

Ina similar fashion, test developers typically present correlations between the 
scores of their tests and those of comparable instruments as evidence of score va- 
lidity. For instance, all of the manuals of major individual intelligence scales cite 
correlations between their scores and those of the other well-established instru- 
ments of the same type. By examining these data one can learn—for example— 
that the correlation between WAIS-II Full Scale IQ and the global composite 
score of the Stanford-Binet-IV (SB-IV), computed for a sample of 26 individuals 
who took both tests, was .88 (Psychological Corporation, 1997, p. 84) or that the 
correlation obtained between the composite scores of the SB-IV and the Kauf- 
man Adolescent & Adult Intelligence Test (KAIT), for a sample of 72 individu- 
als tested with both instruments, was .87 (Kaufman & Kaufman, 1993, p. 100). 
Correlation coefficients of this size are typical for the major intelligence scales 
and serve to corroborate the fact that a good deal of the variance in the scores on 
these tests is shared. 

Correlation coefficients can also be obtained across the scores of subtests 
from different scales. Typical examples of this would be the correlations between 
the scores of various depression scales, for instance, the Depression scale scores 
of the MMPI-2 and those of the Dysthymia scale (r= .68) of the Millon Clinical 
Multiaxial Inventory-II (MCMI-II), or the Beck Depression Inventory scores 
and those of the MCMI-II Major Depression scale (r = .71; Millon, Millon, & 
Davis, 1994, pp. 126, 129). As might be expected, correlation coefficients calcu- 
lated across various types of tests and subtests abound in test manuals and in the 
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psychological literature, even though these indexes often are not very cogent or 
informative. 

Intertest correlations are as ubiquitous as they are because the data for corre- 
lational studies on small convenience samples are easy to gather, especially for 
paper-and-pencil tests that can be administered to groups readily. Correlations 
obtained in this fashion can, of course, range anywhere from zero to +1.00, de- 
pending on the scores in question and nature of the samples used (cf. Chapter 2, 
especially the section on Range Restriction and Correlation). Although the mean- 
ing of any single obtained coefficient is open to interpretation, if sufficient data 
showing consistently high or consistently low correlations across measures are 
accumulated, some patterns of convergence and divergence may be discerned. 
These patterns inform test users about the approximate amounts of shared or 
common variance actoss sets of scores and, indirectly, about the meaning of the 
scores themselves. Consistently high correlations between measures designed to 
assess a given construct—such as the correlations cited in the previous para- 
graph for depression scales—may be taken as evidence of convergent validity, that 
is, evidence regarding the similarity, or identity, of the constructs they are evalu- 
ating. By the same token, discriminant validity evidence, based on consistently low 
correlations between measures that are supposed to differ, also may be used to 
substantiate the identities of the constructs they tap. An example of this kind of 
divergent pattern can be gleaned from the correlations between scores on the 
Bipolar, Manic scale of the MCMI-III and the Depression scale of the MMPI-2 
(r = .06), as well as between scores on the Major Depression scale of the 
MCMI-III and the Hypomania scale of the MMPI-2 (r= .08), both computed on 
a sample of 132 individuals (Millon, Millon, & Davis, 1994, pp. 129-130). 


The Multitrait-Multimethod Matrix 

In an effort to organize the collection and presentation of convergent and dis- 
ctiminant validation data, D. T. Campbell and Fiske (1959) proposed a design 
they called the wultitrait-multimethod matrix (MTMMM). This approach refers to a 
validation strategy that requires the collection of data on two or more distinct 
traits (e.g., anxiety, affiliation, and dominance) by two or more different methods 
(e.g, self-report questionnaires, behavioral observations, and projective tech- 
niques). Once these data are collected and all their intercorrelations are com- 
puted, they can be presented in the form of a matrix, such as the one in Table 5.2. 
Multitrait-multimethod matrices display (a) reliability coefficients for each mea- 
sure, (b) correlations between scores on the same trait assessed by different 
methods (i.e., convergent validity data), and (c) correlations between scores on 
different traits measured by the same methods, as well as (d) between scores on 
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Table 5.2 A Hypothetical Multitrait-Multimethod Matrix (MTMMM) 


Self-Report Observation Projective 
Method Trait Anx Aff Dom Anx Aff Dom Anx Aff Dom 
Anx (.90) 


Self-report Aff 45 (88) 
Dom .35 38 (80 


Anx 60 23 10 (99 
Observation Aff  .25 .58 —08 47 (93) 
Dom 12 —12 .55 30 .32 (86 


Anx .56 22 .11 65 40 31 (94 


Projective Aff 23 57 05 38  .70 29 44° (.89) 
Dom .13 -.10 53 19 26 .68 40 44 (86) 


Note: Anx = Anxiety; Aff = Affiliation; Dom = Dominance. Reliability coefficients are in 
parentheses, along the principal diagonal. Validity coefficients (same trait assessed by dif- 
ferent methods) ate in boldface. All other coefficients are indexes of the discriminant valid- 
ity of scores of different traits assessed by a single method (representing common method 
variance and set in italics) and different traits assessed by different methods (in plain type). 


different traits assessed by different methods (both of which constitute discrim- 
inant validity data). Table 5.2 is a hypothetical MTMMM with a pattern of results 
that would be considered exemplary for this type of validation design. The ma- 
trix in this table shows: 


* the highest coefficients, which are indicative of adequate score reliabil- 
ity (in parentheses), in the principal diagonal; 

* the next highest coefficients—in bold print—between measures of the 
same trait across different methods, indicating convergence among 
their scores; 

* the next highest coefficients—in é/a/ics—between measures of different 

traits assessed by the same method, indicating that a fair amount of the 

variance in the scores is due to the methods employed; and 

the smallest coefficients—in plain type—between measutes of differ- 

ent traits assessed by different methods, indicating that the measures do 

discriminate fairly well among distinct traits. 


The MTMMM design is ideally suited to investigate patterns of convergence 
and divergence among test scores and data gathered from other types of assess- 
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ment instruments. It does, however, constitute a rather stringent validation stan- 
dard that is often difficult to meet, especially for personality assessment instru- 
ments, whose scores ate prone to exhibit a good deal of method variance (i.e., vari- 
ability that is related to characteristics inherent in their methodologies). In 
addition, the MTMMM design is not applied in its full-fledged form very fre- 
quently because collecting information through multiple methods is quite labori- 
ous (see, e.g., Terrill, Friedman, Gottschalk, & Haaga, 2002). Nevertheless, simpler 
variations of the MTMMM scheme, based on the scores of tests that measure both 
similar and dissimilar constructs, albeit through similar methods, are being in- 
creasingly employed in the process of test validation. Furthermore, some instru- 
ments have features that facilitate the collection of data from different sources 
which can then be used to study patterns of convergence and discrimination. For 
example, the Revised NEO Personality Inventory (NEO PI-R) provides parallel 
versions of the same item sets— Form S, for self-reports, and Form R, for ratings 
by observers such as peers or spouses—that can be used to correlate and compare 
scores derived from both sources (Costa & McCrae, 1992, pp. 48—50). 


Age Differentiation 

Test results that are consonant with well-established developmental trends across 
age groups are often seen as evidence of score validity. In fact, the criterion of age 
differentiation is one of the oldest sources of evidence for validating ability tests. 
It may be recalled from Chapter 1 that the success of the original Binet-Simon 
scales was gauged primarily through studies that proved that their sampling of 
cognitive functions produced results that could be used to describe children’s lev- 
els of ability quantitatively, in terms of the age levels to which their performance 
corresponded. In most ability tests, the performance of children and adolescents 
in the normative samples typically shows an upward trend at successive chrono- 
logical ages. At the other end of the age spectrum, declining performance is ob- 
served among samples of older adults on instruments that measure abilities that 
tend to diminish with age, such as memory tests and tests that assess speed of per- 
formance. Age differentiation is also evident in carefully designed studies of 
long-term trends in the performance of individuals at various ages on mental abil- 
ity tests, such as the Seattle Longitudinal Study (Schaie, 1994). Increases or de- 
clines in scores that are consonant with age-appropriate expectations provide ev- 
idence that is necessary, albeit not sufficient, to show that a test is measuring the 
ability constructs it was designed to measure. 


Experimental Results 
Another indirect source of evidence that can be useful in test score validation is 
provided by investigations that use psychological test scores as a dependent vari- 
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able to gauge the effects of experimental interventions. In the area of ability test- 
ing, this evidence derives primarily from pre- and posttest score differences 
following interventions aimed at remediating deficiencies or upgrading perfor- 
mance in various cognitive and intellectual skills. For example, if the scores on a 
test of basic conceptual development in young children (e.g., the Bracken Basic 
Concept Scale-Revised) showed a significant increase for a group exposed to a 
short-term enrichment program—versus no change for a matched group who 
did not participate in the program—the change in scores could be viewed as 
evidence of their validity, as well as of the program’s efficacy. Similar pre- and 
posttest contrasts are often used to document the validity of scores derived from 
personality assessment tools. An example of this type of validation study can be 
found in the manual of the Quality of Life Inventory (QOLI; Frisch, 1994, pp. 
15—16), which is a tool for the assessment of levels of life satisfaction and subjec- 
tive well-being that may be used—among other things—to gauge the effective- 
ness of counseling or psychotherapeutic interventions. 


Factor Analysts 

One way to deal with the huge number of constructs tapped by existing tests— 
and with the unwieldy number of correlations that can be obtained from their 
global scores, their subtest scores, and their item scores—is through a series of 
statistical procedures known collectively as factor analysis (FA). The principal goal 
of factor analysis is to reduce the number of dimensions needed to describe data 
derived from a large number of measures. It is accomplished by a series of math- 
ematical calculations, based on matrix algebra, designed to extract patterns of 
intercorrelation among a set of variables. 

There are two basic ways to conduct factor analyses. The original approach to 
the method is exploratory in nature and thus is known as exploratory factor analysis, 
ot EFA; it sets out to discover which factors (i.e., latent variables or constructs) un- 
derlie the variables under analysis. A second, more recent, approach is called con- 
firmatory factor analysis (CEA) because it sets out to test hypotheses, or to confirm 
theories, about factors that are already presumed to exist. Both approaches can 
be used in analyzing psychological test data, as well as many other kinds of data 
sets. Confirmatory analyses are more sophisticated from the methodological 
standpoint and will be discussed later in this chapter as a subset of the techniques 
for analyzing covariance structures known as structural equation modeling, 
What are the steps involved in factor analyzing psychological test scores? Exploratory factor 
analyses start with a correlation matrix, a table that displays the intercorrelations 
among the scores obtained by a sample of individuals on a wide variety of tests 
(or subtests or items). The fact that this is the starting point of EFA is important 
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in understanding the results of factor analytic research because it points out two 
crucial features of FA that are too often forgotten. Both of these features con- 
cern limitations in the applicability of results stemming from any single FA, 
namely, that the results depend largely on (a) the choice of measures included in 
the analysis and (b) the specific make-up of the sample whose scores provide the 
data for the analysis. 

Table 5.3, Part A, shows an example of a simple correlation matrix. This ma- 
trix was derived from the scores obtained by 95 college students on the five sub- 
tests of the Beta III, which is a nonverbal test of intellectual ability descended 
from the Army Beta (see Chapter 1). The data are part of a study on sex differ- 
ences in cognitive abilities (Urbina & Ringby, 2001). 


Table 5.3 


A. Correlation Matrix: Intercorrelations of Scores on Five Subtests of the Beta III for 95 
College Students 


Picture Clerical Picture Matrix 
Subtest Coding Completion Checking Absurdities Reasoning 
Coding 1.00 .13 .62** .20 .05 
Picture Completion 1.00 .09 21* 1 
Clerical Checking 1.00 18 .20 
Picture Absurdities 1.00 31** 
Matrix Reasoning 1.00 


Note: Data ate from Urbina and Ringby (2001). 
*p < .05 **p < .01 


B. Factor Matrix for the Two Factors Extracted from the Exploratory Factor Analysis 
(EFA) of Five Beta III Subtests* 


Subtest Loadings on Factor 1 Loadings on Factor 2 
Coding 90 -06 
Picture Completion 07 54 
Clerical Checking .88 14 
Picture Absurdities 15 .75 
Matrix Reasoning .02 .73 


Note: Boldface indicates the highest loadings on the two varimax-rotated factors. 


“Factors 1 and 2 account for 61% of the variance in the subtest scores; the remaining 39% 
of the variance is accounted for by factors specific to each subtest, and by error variance. 
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The next steps in factor analysis depend on the specific choice of techniques 
employed by the investigator. Several different procedures may be used to con- 
duct these analyses and to extract factors (see, e.g., Bryant & Yarnold, 1995; Com- 
rey & Lee, 1992). A discussion of factor analytic procedures is beyond our pur- 
poses, as it would involve delving into technicalities, such as methods for 
extraction and rotation of factors, that are fairly complex. Nevertheless, the fact 
that various approaches to FA exist must be noted and kept in mind because one 
can arrive at different solutions depending on the assumptions and methods that 
are used. Differences in the solutions generally pertain to the number of factors 
extracted and to their relative independence from one another. For insight into 
these and other issues related to the methodology of FA, see Russell (2002). 

The end product of factor analyses is a factor matrix, which is a table that lists 
the loadings of each one of the original variables on the factors extracted from 
the analyses. Factor loadings are correlations between the original measures in the 
correlation matrix and the factors that have been extracted. Part B of Table 5.3 
displays the factor matrix for the two factors extracted from a factor analysis of 
the data in the correlation matrix in Part A of the same table. This factor matrix 
indicates that two subtests (Coding and Clerical Checking) have very high load- 
ings on Factor 1 and negligible loadings on Factor 2, whereas the other three sub- 
tests (Picture Completion, Picture Absurdities, and Matrix Reasoning) show the 
reverse pattern in their factor loadings. 

Interpreting the results of factor analyses. Once factor matrices are obtained, they can 
be examined to try to determine the nature of the factors that account for most 
of the variance in the original data set. The factors themselves are then labeled on 
the basis of inductive logic. In order to identify and label the factors, one exam- 
ines the distinguishing features of the measures that load most and least heavily 
on each of the factors in the matrix. In our example, the factor matrix in Table 5.3 
suggests that the first factor involves speed of performance, because the two subtests 
that load heavily on that factor involve extremely simple tasks and have very brief 

time limits. The second factor has 

high loadings on the remaining three 


DON’T FORGET subtests, all of which involve prob- 


Pereeeeee eee eee eee eee eee reece eee eee ree errr eee eee ee eee eee eee eee eee lems that require reasoning based on 
Factors are not “real” entities, al- 
though they are often discussed as 
though they were. They are simply 


pictorial or graphic stimuli. This fac- 
tor matrix pattern coincides with the 


constructs or latent variables that may ones derived from exploratory and 
be inferred from the patterns of co- confirmatory factor analyses of the 
variance revealed by statistical analy- 


Boe Beta III standardization sample data, 


which are presented in the Beta II 
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manual. Factors 1 and 2 are labeled “Processing Speed” and “Nonverbal Reason- 
ing,” respectively (Kellogg & Morton, 1999). 

Factor analysis was developed by psychologists in an attempt to investigate the 
underlying bases for the interrelationships among test scores, and among the 
scores of various types of ability tests in particular. However, the techniques of 
FA were promptly applied to personality test data and personality trait descrip- 
tions as well. The history of FA, both in the area of abilities as well as personality, 
has been fraught with controversy about the appropriateness of its various meth- 
ods and the extrapolations that can reasonably be made on the basis of factor an- 
alytic results (Cowles, 2001, chap. 11). 

In the field of cognitive abilities, much of the controversy has centered on the 
general factor of mental ability, or g (originally postulated by Charles Spearman), 
especially on questions related to its significance and heritability (Jensen, 1998). 
Additional theorizing and basic research in this field have dealt mainly with issues 
concerning the nature, number, and organization of intellectual traits. An out- 
standing compilation of much of the factor analytic literature on human cogni- 
tive abilities, along with a widely accepted hierarchical theory of cognitive trait or- 
ganization, can be found in John Carroll’s (1993) book on this subject. 

In the field of personality assessment, factor analysis has been applied to the 
task of identifying and measuring the major dimensions required for a compre- 
hensive description of personality, an issue about which there has also been a 
good deal of debate. Within this area, two separate traditions of factor analytic re- 
search arose independently. One of them centered from the outset on the use of 
personality questionnaire data. The other—known as the /exical tradition— 
started by reducing the myriad of words used to describe personality attributes to 
a manageable number by combining synonyms. This was followed by an attempt 
to identify the primary dimensions of personality through intercorrelations and 
factor analyses of ratings on various traits assigned to heterogeneous groups of 
individuals by their associates, as well as by self-report questionnaire data. More 
recently, research from both traditions has coalesced and achieved some degree 
of consensus. The prevailing model centers on the use of a hierarchical pattern of 
analysis to simplify the collection of data of varying degrees of generality perti- 
nent to personality functioning, and has come to be known as the five-factor model 
(FFM; Carroll, 2002; Costa & McCrae, 1992; Digman, 1990; Wiggins & Pincus, 
1992). 

In spite of its problems and limitations, the factor analytic tradition has been 
extraordinarily fruitful for psychological testing and, more generally, for psycho- 
logical theorizing. The longevity of these methods, as well as their continuing re- 
finements and extensions, have cteated a tich archive of tools and data from 
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which to proceed to extend our understanding of psychological traits and tests. 
Rapid Reference 5.5 outlines some of the benefits that can be derived from FA, 
as well as its major limitations. 


Structural Equation Modeling Techniques 

Exploratory factor analysis is just one of several types of multivariate statistical 
techniques that allow investigators to examine the relations among multiple mea- 
sures to try to determine the underlying constructs that account for observed 
variability. The increased availability of powerful computers and computer soft- 
ware in recent decades has greatly enhanced the ease with which factor analytic 
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Part I: Benefits of Factor Analysis 


Construct validation: By pooling together large numbers of measures and examin- 
ing the factors that seem to account for the variance those measures share, we 
can learn more about the composition of tasks sampled by psychological tests 
and about the organization of traits, in terms of their generality and specificity. 


Practical application: When a test battery is made up of a large number of tests, 
factor analytic results provide a way to simplify the interpretation and reporting of 
subtest scores. This is accomplished by means of factor scores, which are essen- 
tially indexes that aggregate subtest scores into a smaller number of cohesive con- 
struct categories derived from factor analyses. 


Part Il: Limitations of Factor Analysis 


Interpretation of the results of any single factor analytic study cannot extend be- 
yond the data used in the analysis, in terms either of what is being measured or of 
their generalizability across populations. 


What is being measured? Both the manual of the Beta Ill (a test that purports to 
measure nonverbal intellectual ability) and the analysis displayed in Table 5.3 sug- 
gest that the five subtests of the Beta Ill can be configured into two clusters that 
have a good deal of variance in common. Examination of the tasks involved in the 
five subtests confirms that the two factors are nonverbal. However, these data 
cannot reveal whether or to what extent these factors capture the essential as- 
pects of nonverbal intellectual ability. 


Generalizability of factor analytic results: The correlational data derived from 95 col- 
ege students (Urbina & Ringby, 2001) and presented in Table 5.3 yield results sim- 
ilar to those obtained with the Beta Ill standardization group, a much larger (N = 
|,260), and far more representative, sample of the U.S. population. Although this 
convergence of results supports the factor structure obtained in both investiga- 
tions, it leaves open the question of whether this structure would generalize to 
other populations, such as people of different cultures, individuals with uncor- 
rected hearing or visual impairments, or any other group whose experiential his- 
tory differs significantly from that of the samples used in the two analyses at hand. 
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techniques—and other sophisticated methods of analysis of multivariate corre- 
lational data, such as multiple regression analyses—can be used to investigate la- 
tent variables (i.e., constructs) and the possible direct and indirect causal links or 
paths of influence among them. 

One rapidly evolving set of procedures that can be used to test the plausibility 
of hypothesized interrelationships among constructs as well as the relationships 
between constructs and the measures used to assess them is known as structural 
equation modeling (SEM). The essential idea of all SEM is to create one or more 
models—based on theories, previous findings, or prior exploratory analyses—of 
the relations among a set of constructs or latent variables and to compare the co- 
variance structures or matrices implied by the models with the covariance matri- 
ces actually obtained with a new data set. In other words, the relationships ob- 
tained with empirical data on variables that assess the various constructs (factors 
or latent variables) are compared with those predicted by the models. The corre- 
spondence between the data and the models is evaluated with statistics, appro- 
ptiately named goodness-of-fit statistics. SEM provides several advantages over 
traditional regression analyses. Its advantages derive primarily from two charac- 
teristics of this methodology: (a) SEM is based on analyses of covariance structures 
(.e., patterns of covariation among latent variables or constructs) that can repre- 
sent the direct and indirect influences of variables on one another, and (b) SEM 
typically uses multiple indicators for both the dependent and independent vari- 
ables in models and thus provides a way to account for measurement error in all 
the observed variables. Readers who wish to pursue the topic of SEM, and related 
techniques such as path analysis, in more detail might consult one or more of the 
sources suggested in Rapid Reference 5.6. 

As fat as psychological test validation is concerned, SEM techniques are used 
for the systematic exploration of psychological constructs and theories through 
research that employs psychological testing as a method of data collection for one 
or more of the indicators in a model. SEM can provide supporting evidence for 
the reliability of test scores as well as for their utility as measures of one or more 
constructs in a model. However, at present, the most extensive application of 
SEM techniques in test score validation is through confirmatory factor analyses. 

Confirmatory factor analysis (CEA), mentioned briefly in a preceding section, in- 
volves the a priori specification of one or more models of the relationships be- 
tween test scores and the factors or constructs they are designed to assess. In 
CFA, as in all other SEM techniques, the direction and strength of interrelation- 
ships estimated by various models are tested against results obtained with actual 
data for goodness of fit. These analyses have been facilitated by the development 
of computer programs—such as LISREL (Joreskog & Sérbom, 1993)—de- 
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= fid Reference 5.6 


Sources of Information on 
Structural Equation Modeling (SEM) 


e Fora good introduction to SEM that does not presuppose knowledge of statis- 
tical methods beyond regression analysis, see the following: 


Raykov, T., & Marcoulides, G. A. (2000). A first course in structural equation model- 
ing. Mahwah, NJ: Erlbaum. 


e Amore advanced presentation of SEM that includes contributions from many 
of the leading authorities on this methodology can be found in the following: 


ollen, K. A., & Long, J. S. (Eds.). (1993). Testing structural equation models. New- 
ury Park, CA: Sage. 


he Internet is an excellent source of information about many topics, including 
EM techniques that are being used in a number of fields. One of the best 
starting points is a site created and maintained by Ed Rigdon, a professor in the 
arketing Department of Georgia State University. The address for this site is 
as follows: 


e 
WS) oe 


http://www.gsu.edu/~mkteer/sem.html. 


signed to generate values for the hypothesized models that can then be tested 
against the actual data. 

Examples of this kind of work are becoming increasingly abundant both in the 
psychological literature and in the validity sections of test manuals. The confir- 
matory factor analyses conducted with the standardization sample data of the 
WAIS-HI (Psychological Corporation, 1997, pp. 106—110) are typical of the 
studies designed to provide validity evidence for the psychological test scores. In 
those analyses, four possible structural models—a two-factor, a three-factor, a 
four-factor, and a five-factor model—were successively evaluated and compared 
to a general, one-factot model to determine which of them provided the best fit 
for the data in the total sample and in most of the age bands of the WAIS-III nor- 
mative group. The results of the CFA indicated that the four-factor model pro- 
vided the best overall solution and confirmed the patterning previously obtained 
with EFAs of the same data. These results, in turn, wete used as the bases for de- 
termining the composition of the four index scores (Verbal Comprehension, Per- 
ceptual Organization, Working Memory, and Processing Speed) that can serve to 
organize the results of 11 of the 14 WAIS-III subtests into separate domains of 
cognitive functioning. 

In contrast to the kind of CFA conducted with the WAIS-III data, other CFA 
studies involve more basic work aimed at clarifying the organization of cognitive 
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and personality traits. An example of this type of study can be found in Gustafs- 
sons (2002) description of his reanalyses of data gathered by Holzinger and 
Swineford in the 1930s from a group of seventh- and eighth-grade students (N = 
301) who were tested with a battery of 24 tests designed to tap abilities in five 
broad ability areas (verbal, spatial, memory, speed, and mathematical deduction). 
Using two different models of CFA, Gustafsson found support for the hypothe- 
sized overlap between the G (general intelligence) and Gf (fluid intelligence or 
reasoning ability) factors that had been suggested by the analyses done in the 
1930s. Gustafsson also used contrasts in the patterns of results from the original 
factor analysis done in the 1930s and his own contemporary CFAs to illustrate 
significant implications that various measurement approaches, as well as sample 
composition, have for the construct validation of ability test scores. 

Confirmatory factor analysis and other structural modeling techniques are still 
evolving and are far from providing any definitive conclusions. However, the 
blending of theoretical models, empirical observations, and sophisticated statis- 
tical analyses that characterizes these techniques offers great promise in terms of 
advancing our understanding of the measures we use and of the relationships be- 
tween test scores and the constructs they are designed to assess. 


Validity Evidence Based on Relationships 
Between Test Scores and Criteria 


If the goals of psychological testing were limited simply to describing test takers’ 
performance in terms of the frames of reference discussed in Chapter 3 or to in- 
creasing our understanding of psychological constructs and their interrelation- 
ships, the sources of evidence already discussed might suffice. However, the valid 
interpretation of test scores often entails applying whatever meaning is inherent 
in scores—whether it is based on norms, test content, response processes, es- 
tablished patterns of convergence and divergence, or any combination of these 
soutces of evidence—to the pragmatic inferences necessary for making deci- 
sions about people. When this is the case, validity evidence needs to address the 
significance test scores may have in matters that go beyond those scores or in 
realms that lie outside the direct purview of the test. In other words, test scores 
must be shown to correlate with the various criteria used in making decisions and 
predictions. 


Some Essential Facts About Criteria 
Merriam-Webster’s Collegiate Dictionary (1995) defines criterion as “a standard on 
which a judgment or decision may be based” or “a characteristic mark or trait.” 
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Although the plural form, criteria, is often used as a singular, in the present con- 
text it is necessary to apply both forms of the word appropriately because they are 
both central to our purposes. A criterion may also be defined, more loosely, as 
that which we really want to know. This last definition, though less formal than the 
dictionarty’s, highlights the contrast between what test scores tell us and the prac- 
tical reasons why we use tests. 

For psychological tests that are used in making judgments or decisions about 
people, evidence of a relationship between test scores and criterion measutes is 
an indispensable, but not necessarily sufficient, basis for evaluating validity. C7i- 
terion measures are indexes of the criteria that tests are designed to assess or predict 
and that are gathered independently of the test in question. Rapid Reference 5.7 
provides a list of the kinds of criteria typically used in validating test scores. Since 
the nature of criteria depends on the questions one wishes to answer with the help 
of tests, it follows that validation procedures based partly or entirely on relation- 
ships between test scores and criterion measures must produce evidence of a link 
between the predictors (test scores) and the criteria. 

Criterion measures or estimates may be naturally dichotomous (e.g., graduating 
vs. dropping out) or artificially dichotomized (e.g., success vs. failure); po/ytomous 
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Typical Criteria Used in Validating Test Scores 


Although there is an almost infinite number of criterion measures that can be em- 
ployed in validating test scores, depending on what the purposes of testing are, 
the most frequent categories are the following: 


e Indexes of academic achievement or performance in specialized training, such as 
school grades, graduation records, honors, awards, or demonstrations of com- 
petence in the area of training through successful performance (e.g., in piano 
playing, mechanical work, flying, computer programming, bar exams, or board 
certification tests). 


e Indexes of job performance, such as sales records, production records, promo- 
tions, salary increases, longevity in jobs that demand competence, accident-free 
job performance, or ratings by supervisors, peers, students, employees, cus- 
tomers, and so forth. 


e Membership in contrasted groups based on psychiatric diagnoses, occupational 
status, educational achievement, or any other relevant variable. 


e Ratings of behavior or personality traits by independent observers, relatives, 
peers, or any other associates who have sufficient bases to provide them. 


e Scores on other relevant tests. 
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(e.g., diagnoses of anxiety vs. mood vs. dissociative disorders, or membership in 
artistic vs. scientific vs. literary occupations); or continuous (e.g., grade point aver- 
age, number of units sold, scores on a depression inventory, etc.). Whereas the na- 
ture of criteria depends on the decisions or predictions to be made with the help 
of test scores, the methods used to establish the relationships between test scores 
and criteria vary depending on the formal characteristics of both test scores and 
criterion measures. In general, when the criterion measure is expressed in a di- 
chotomous fashion (e.g., success vs. failure) or in terms of a categorical system 
(e.g. membership in contrasted groups), the validity of test scores is evaluated in 
terms of Ait rates. Hit rates typically indicate the percent of correct decisions or 
classifications made with the use of test scores, although mean differences and 
suitable correlation indexes may also be used. When criterion measures are con- 
tinuous (e.g., scores on achievement tests, grades, ratings, etc.) the principal tools 
used to indicate the extent of the relationship between test scores and the crite- 
rion measure ate correlation coefficients. However, if a certain value on a con- 
tinuous criterion, such as a 2.0 grade point average, is used as a cutoff to deter- 
mine a specific outcome, such as graduation from college, scores on the predictor 
test can also be evaluated in terms of whether they differentiate between those 
who meet or exceed the criterion cutoff and those who do not. 

The history of psychological testing in recent decades reflects not only an evo- 
lution in understanding the nature and limitations of tests and test scores but also 
an increased appreciation of the significance and complexity of criterion mea- 
sures (see, e.g., James, 1973; Tenopyr, 1986; Wallace, 1965). As a result, with rare 
exceptions, the notion that there is such a thing as “a criterion” against which a 
test may be validated is no longer any more tenable than the proposition that the 
validity of a test can be determined on an all-or-none basis. Instead, the follow- 
ing facts about criteria are now generally understood: 


1. In most validation studies, there are many possible indexes (both 
quantitative and qualitative) that can qualify as criterion measures, in- 
cluding scores from tests other than those undergoing validation. 
Therefore, careful attention must be given to the selection of criteria 
and criterion measures. 

2. Some criterion measures are more reliable and valid than others. Thus, 
the reliability and validity of criterion measures need to be evaluated, 
just as the reliability and validity of test scores do. 

3. Some criteria are more complex than others. As a result, there may or 
may not be a correlation among criterion measures, especially when 
criteria are multifaceted. 
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4. Some criteria can be gauged at the time of testing; others evolve over 
time. This implies that there may or may not be substantial correla- 
tions between criterion measures that ate available shortly after testing 
and more distant criteria that may be assessed only over a longer pe- 
riod of time. 

5. Relationships between test scores and criterion measures may or may 
not generalize across groups, settings, or time periods. Therefore, 
ctiterion-related validity evidence needs to be demonstrated anew for 
populations that differ from the original validation samples in ways 
that may affect the relationship between test scores and criteria, as well 
as across various settings and times. 

6. The strength or quality of validity evidence with regard to the assess- 
ment or prediction of a criterion is a function of the characteristics of 
both the test and the criterion measures employed. If the criterion 
measures ate unreliable or arbitrary, indexes of test score validity will 
be weakened, regardless of the quality of the test used to assess or pre- 
dict the criteria. 


Criterion-Related Validation Procedures 


The criterion-related decisions for which test scores may be of help can be clas- 
sified into two basic types: (a) those that involve determining a person’s current 
status and (b) those that involve predicting future performance or behavior. In a 
sense, this dichotomy is artificial because regardless of whether we need to know 
something about a person’s current status or future performance, the only infor- 
mation test scores can convey derives from current behavior—that is, from the 
way test takers perform at the time of testing. Nevertheless, criterion-related val- 
idation procedures are frequently categorized as either concurrent or predictive, 
depending on the measures employed as well as their primary goals. 


Concurrent and Predictive Validation 

Concurrent validation evidence is gathered when indexes of the criteria that test 
scores are meant to assess are available at the time the validation studies are con- 
ducted. Strictly speaking, concurrent validation is appropriate for test scores that 
will be employed in determining a person’s current status with regard to some 
classificatory scheme, such as diagnostic categories or levels of performance. Pre- 
dictive validation evidence, on the other hand, is relevant for test scores that are 
meant to be used to make decisions based on estimating future levels of perfor- 
mance or behavioral outcomes. Ideally, predictive validation procedures require 
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collecting data on the predictor variable (test scores) and waiting for criterion 
data to become available so that the two sets of data can be correlated. This pro- 
cess is often impractical because of the time element involved in waiting for cri- 
teria to mature and also because of the difficulty of finding suitable samples to use 
in such studies. As a result, concurrent validation is often used as a substitute for 
predictive validation, even for tests that will be used for estimating future perfor- 
mance, such as college admissions or preemployment tests. In these cases, the test 
under development is administered to a group of people, such as college sopho- 
mores or employees, for whom criterion data are already available. 

Often the distinction between the two kinds of validation procedures hinges 
on the way test users pose the questions they want to answer with the help of a 
test. Rapid Reference 5.8 contains examples of some typical questions and 
decision-making situations that may call for either concurrent or predictive vali- 
dation evidence, depending on how the question is posed and on the time hori- 
zon that is chosen. To illustrate the distinction between concurrent and predic- 
tive validation strategies, a relatively simple example of each type of study will be 
presented, followed by a discussion of the major issues pertinent to criterion- 
related validation. 


Concurrent Validation Example: The Whitaker Index 

of Schizophrenic Thinking 

Tests that are used to screen for psychiatric disorders, such as schizophrenia or 
depression, usually undergo concurrent validation. Typically, these studies em- 
ploy two or more samples of individuals who differ with respect to their inde- 
pendently established diagnostic status. One of the many instruments whose 
scores are validated in this fashion is the Whitaker Index of Schizophrenic Think- 
ing (WIST; Whitaker, 1980). The WIST was designed to identify the kind of 
thinking impairment that often accompanies schizophrenic syndromes. Each of 
its two forms (A and B) consists of 25 multiple choice items. 

In standardizing the WIST, Whitaker used samples of acute and chronic schiz- 
ophrenic patients (S), as well as three groups of nonschizophrenics (NS), to de- 
tive cutoff scores that would optimally differentiate S from NS individuals. The 
cutoff scores established with the standardization groups discriminated between 
S and NS groups with 80% efficiency for Form A and 76% efficiency for Form 
B. Thus, depending on the form, the cutoff index resulted in 20% to 24% incor- 
rect decisions. With Form A, incorrect decisions of the false negative type—those 
in which S subjects were classified as NS by the index—were much higher (33%) 
than false positive decisions, wherein NS subjects were classified as S (10%). The 
same pattern (38% false negatives vs. 13% false positives) was obtained with 
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The Relationship Among Questions, Decisions, and 
Predictions Requiring Criterion-Related Validation 


Questions About 
Current Status 


Immediate Goal: 
Decisions 


Implicit Questions: 
Predictions 


Is John X suffering from 

Schizophrenia, Panic Dis- 
order, Clinical Depression, 
Attention Deficit, or 
some other mental dis- 
order? 


Is Mary Y subject to sui- 
cidal (or homicidal) im- 
pulses she may not be 
able to overcome if left 
to her own devices? 


Is Joe Z gifted (or suffer- 
ing from severe mental 
retardation)? 


Is Tom P honest and trust- 
worthy (or motivated for 
sales work)? 


Is Jane Q capable of doing 
college-level work (or 
flying an airplane)? 


Should John X receive the 
treatment (medication, 
psychotherapy, etc.) that 
is recommended for the 
disorder in question? 


Should Mary Y continue 
to be hospitalized (or in- 
carcerated)? 


Should Joe Z be admitted 
to a special education 
program for gifted (or 
mentally retarded) indivi- 
duals? 


Should Tom P be hired 
for work as a cashier 
(or salesperson)? 


Should Jane Q be admit- 
ted into college (or 
granted an airplane pilot's 
license)? 


Will John X profit (a little, 
a lot, or not at all) from 
the recommended treat- 
ment? 


Will Mary Y attempt to 
kill herself (or someone 
else) if left at liberty? 


Will Joe Z benefit (a little, 
a lot, or not at all) from a 
special education pro- 
gram? 


Will Tom P be a consci- 
entious cashier (or a suc- 
cessful salesperson), or 
will he steal money (or 
make few sales)? 


Will Jane Q be able to 
finish college with a 
grade-point average high 
enough to graduate (or be 
able to fly a plane with- 
out causing accidents)? 


Form B. See Chapter 7 for further explanation of the terminology used to desig- 
nate incorrect decisions (e.g., false positive and false negative decisions). 
Additional validation evidence for the WIST includes studies done in Mexico 
and Spain with Spanish translations of the instrument. These studies show some 
support for the WIST’s ability to detect the thought impairment associated with 
schizophrenia, albeit with different rates of efficiency and different cutoff scores. 
The hit rate obtained when the WIST cutoff scores are used to discriminate S 
from NS subjects is fairly typical for tests of this kind. Because of the relatively 
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high error rates that the WIST cutoff z = 3 
scores can yield, they should never be CAUTION 


used as the sole vehicle for establish- p : 
e Psychological test results, as well as 


ing a diagnosis of schizophrenia, any medical test results and those in 


more than any other single indicator many other fields, are not expected 


should. However, depending on the to achieve (00e accunacy. 


e Because of the less-than-perfect na- 
ture of all diagnostic indicators, psy- 
chologists engaged in clinical assess- 

screening battery or as one index of ment—as well as diagnosticians in 

other fields—should never rely on 

a single indicator. 


setting and context of testing, the 
WIST may prove useful as part of a 


change in the symptomatology of pa- 
tients diagnosed with schizophrenia. 
Although the discriminations made 
with the use of WIST cutoff scores are far from perfect, they may still be useful 
in investigating the possibility that a person suffers from thought impairment. 
For a review of the WIST, see Flanagan (1992). 


Predictive Validation: A Hypothetical Example 
Tests that are used to predict performance in occupational and educational set- 
tings require validation strategies geared toward prediction. The ideal design for 
a predictive validation study in either one of those fields would involve the fol- 
lowing steps: (a) testing an unselected group of applicants with an ability test or 
test battery, (b) hiring or admitting them without regard to their test scores, (c) 
waiting until criterion measures of job or academic performance became avail- 
able, (d) obtaining correlations between the preemployment or admission test 
scores and criterion measures, and (e) using the correlational data to derive a re- 
gression equation to estimate or predict the criterion performance of future ap- 
plicants. For obvious reasons, most employers and school administrators are not 
willing to hire or accept all applicants—especially when their numbers exceed the 
number of positions available—in order to conduct a validation study. Never- 
theless, for the sake of simplicity and to illustrate the way in which predictions can 
be made on the basis of test scores, a simple hypothetical example of this type of 
study will be described. 

To provide some context for this example, let us say that—in order to ensure 
a profitable level of production—the owner of a small factory wants to hire 
people who are able to process at least 50 widgets per hour on the assembly line. 
Since there are no other requirements for this job, besides showing up for work 
on time, the criterion to be predicted is simply the number of widgets produced 
per hour on the assembly line. Output on the assembly line is primarily a function 
of the workers’ speed and accuracy in manual tasks. Therefore, a test of manual 
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Table 5.4 Data for Predictive Validation Example 


X-M, Y-M, 


Applicant Test Score (X) Output (Y) (x) (y) x? y xy 

1 18 56 5 6 25 36 30 

2 12 50 —1 0 1 0 0 

3 8 47 —5 -3 25 9 815 

4 20 52 7 2 49 4 14 

5 14 52 1 2 1 4 2 

6 5 42 -8 -8 64 64 64 

a 10 48 -3 -2 9 4 6 

8 12 49 —1 —1 1 1 1 

9 16 50 3 0 9 0 0 

10 15 54 2 4 4 16 8 

Sum 130 500 0 0 188 138 140 
Mean 13 50 


dexterity is selected as the predictor. Table 5.4 shows the bivariate data for 10 hy- 
pothetical applicants for the assembly line job who take the manual dexterity test, 
are hired and trained, and have their assembly line output per hour gauged on repeated 
occasions and averaged so as to produce a viable criterion measure. 

As discussed in Chapter 2, when two variables exhibit a linear relationship and 
a strong correlation with each other, it is possible to predict one based on knowl- 
edge of the other by applying the linear regression model. Figure 5.1 shows the 
scatter diagram for the bivariate data in Table 5.4, and indicates that the relation- 
ship between the manual dexterity test scores and assembly-line output per hour 
is strong, positive, and linear. 

In this case, it is necessary to solve a linear regression equation to predict the 
criterion of assembly-line output (the Y variable) from scores on the manual dex- 
terity test (the X variable). This equation expresses the relationship between X 
and Y, and contains the two major components needed to draw the regression line, 
portrayed in Figure 5.1. The regression line is the line that best fits the bivariate 
data, in that it minimizes errors in predicting Y from X. The two pivotal compo- 
nents of the linear regression equation are (a) the Y intercept, which is the point 
at which the line meets the vertical axis representing the Y variable, and (b) the 
slope of the line, which is the ratio of change in the Y variable for every unit of 
change in the X variable. These values, as well as the correlation coefficient for X 
and Y, are computed from the bivariate data. The necessary analyses have been 
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Assembly Line Output per Hour 


4 6 8 10 12 14 16 18 20 22 


Manual Dexterity Test Scores 


Figure 5.1 Scatter diagram of data and regression line for predictive valida- 
tion example 


performed in Table 5.5. This table shows that the Pearson 7 (7) is .87 (significant 
at the .001 level). The coefficient of determination, obtained by squaring ris .755. It 
indicates that 75.5% of the variance in Y (assembly line output) is associated with 
the variance in X (scores on the dexterity test). Both of these coefficients confirm 
that there is a strong relationship between X and Y, making it possible to predict 
the criterion measure based on the test scores with substantial accuracy. The fac- 
tory owner in our example may apply this information to estimate the assembly 
line output of subsequent applicants by administering the manual dexterity test 
and inserting their scores in the regression equation, as follows. 

Suppose that the 11th person to apply for a job (after the 10 who were used in 
the hypothetical validation study) obtains a manual dexterity test score of 17. Us- 
ing the coefficients shown in Table 5.5 to solve the regression equation Y’ = 40.32 
+ .745 (17), the factory owner would learn that the estimated assembly line out- 
put per hour for the 11th applicant is 53 widgets, or three more than the desired 
minimum number. 

Correlations between test scores and criterion measures (r,.) are usually called 
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Table 5.5 Analysis of Predictive Validation Data 


Number of Standard 
Descriptive Statistic Observations (V) Mean Deviation (SD) 
X = predictor (test scores) 10 13 4.57 
Y = criterion (output) 10 50 3.91 
x xy 140 
Pearson r r = g - = = 87 
* (N= )SD)SD) — (9)(4.57)8.91) 
Coefficient of determination r2 =.755 
Linear regression equation’ Y’ = a, + b AX) 
Example data a, = 40.32 


b= .745 
Regression equation for predicting assembly-line output based on dexterity test scores 
Predicted output = Y’ = 40.32 + .745 (X) 


*Where Y’ = predicted criterion score; a , = M, — b (M) = intercept of the regression line; 
b= (229) /(2 x”) = slope of the regression line; and X = score on the predictor (manual 
dexterity test score). 


validity coefficients. If r. were 1.00, indicating a perfect correlation between test 
scores and the criterion, perfect prediction would be possible. Although the r of 
87 obtained with the data in our example is extremely high—much higher than 
the typical predictive validity coefficients obtained with real data—it is not per- 
fect. Thus, there is bound to be some error in the estimates of the criterion made 
by using the test scores. This error is gauged by the standard error of estimate (SE), 
a statistic that expresses, in the scale used for the criterion measure, the error in 
predictions that are based on imperfect correlations. 

Formulas for the SE „and for a correction term applied to the SE.,, due to the 
small sample size are presented in Rapid Reference 5.9, along with the results for 
the data in our example. Interpretation of the SE „assumes (a) that the predicted 
criterion score, or Y’, is the average value in a hypothetical normal distribution of 
all possible criterion scores for the applicant in question, and (b) that the SE „is 
the standard deviation of that distribution. These assumptions allow us to attach 
a level of probability to the predictions made from test scores, based on the nor- 
mal curve ateas in Appendix C. To do this, one calculates the range encompassed 
within Y’ + SE... (z), where z is the value that corresponds to the desired proba- 
bility level. For the 11th applicant, in our example, the dexterity test score of 17 
resulted in a predicted Y’ of 53. The SE... for the test scores, calculated in Rapid 
Reference 5.9, is 2.05 (or 2). Therefore, we can predict that the chances that this 


applicant will produce between 51 and 55 widgets per hour (i.e., 53 + 2) on the as- 
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DON’T FORGET 


Evaluating the adequacy of validity coefficients of various magnitudes is a relative 
matter that depends on the purpose for which test scores will be used. In general, 
there are two major ways to approach this matter: 


|. When validity is expressed in the form of a correlation coefficient (r,,), the 
proportion of variance shared by the predictor (X) and the criterion (Y) is of- 
ten estimated by squaring r, and obtaining the coefficient of determination, or 
rž. A coefficient of determination, discussed in Chapter 2, expresses the pro- 
portion of the variance in Y that is associated with the variance in X. For ex- 
ample, the predictive validity coefficient of .87 obtained in the example pre- 
sented in Table 5.5 indicates that the scores on the manual dexterity test used 
as a predictor can explain approximately 76% of the variance (.87 x .87 = .76) 
in the criterion of assembly line output. 


2. Validity coefficients can also be evaluated in terms of the incremental validity 
contributed by a test—that is, by the extent to which use of a test (or any 
other instrument) brings about an increase in the efficiency of decisions made 
in a given situation compared to the validity of other methods. This aspect of 
validity concerns the utility of a test and is discussed further in Chapter 7. 


Coefficients in the .20s and .30s are not uncommon in predictive validity studies. 
As a general rule, coefficients of 40 or higher are considered acceptable. Al- 
though some of these figures may seem low, they must be viewed in the context 
of the multidetermined nature of performance in most endeavors and they must 
be weighed in light of the predictive efficiency of alternative methods of making 
selection decisions. 


= fid Reference £9 


Standard error of estimate (SE...) 
Se = DL ae 
SE.., for predictive validation example: 
SE... = 3.91 V1 — 755 = 3.91(.495) = 1.935 


SE corrected for sample size: 


[N= | Mo-i 
E E a = | ees = RENO ITA 


sembly line are 68/100. To make a prediction at the 95% (instead of 68%) confi- 
dence level we would use a z value of 1.96, which dematcates the .05 level of sig- 
nificance fora two-tailed test. Solving the equation Y’ + SE, (1.96), we could pre- 
dict that chances are 95/100 that the applicant in question would produce 
between 49 and 57 widgets per hour (53 + 4). 
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Issues Concerning Criterion-Related Validation Studies 


Some of the difficulties inherent in criterion-related validation studies were men- 
tioned earlier in this chapter in connection with the discussion of the notion of 
criteria. Additional considerations that deserve thorough examination by poten- 
tial users of instruments buttressed by this kind of validity evidence are discussed 
next, using some of the features of the examples just described as a starting point. 
Although our scope does not allow extensive exploration of these issues—or of 
the various methods that can be used to deal with them—some of the more 
salient ones need to be briefly described. 


Characteristics of Criterion Measures 
As mentioned earlier, the criteria upon which test scores are validated can differ 
greatly in terms of their own reliability and validity. In the case of the WIST, the 
criterion used in establishing the validity of scores was the diagnostic status (S vs. 
NS) of the individuals in the validation samples. If the initial classification of sub- 
jects in these studies included some incorrect diagnoses (some S’s who were NS’s, 
ot viceversa), validity data would obviously be weakened. A similar caution ap- 
plies to all validation procedures that rely on subjective criteria, such as ratings or 
other qualitative judgments that are used to categorize people into criterion 
groups. In addition, the validity of criterion measures can be eroded when those 
who are charged with determining the criterion standing of individuals in the val- 
idation samples have access to scores on the test that is used as a predictor. This 
type of error, known as criterion contamination, is easily prevented by making sure 
that teachers, supervisors, diagnosticians, and others who assign ratings or make 
judgements related to criterion measures remain unaware of and are not influ- 
enced by the knowledge of test scores. With regard to the ratings themselves, as 
well as other criterion measures that depend on subjective judgment, test devel- 
opets need to provide evidence that the instruments and methods used to rate or 
to classify the criterion groups employed in validation studies are reliable and 
valid. When the criterion consists of membership in a group such as a certain di- 
agnostic category, its reliability and validity can be improved by careful subject se- 
lection that is based on evidence from multiple and preferably independent 
sources. As far as ratings criteria are concerned, their reliability and validity 
should also be ascertained. To this end, there is an extensive literature devoted to 
ways of training raters so as to minimize biases and to exploring the best rating 
formats and methodologies (see, e.g., Guion, 1998, chap. 12). 

Our hypothetical study of the validity of manual dexterity test scores—meant 
to illustrate predictive validation procedures as simply as possible—contains sev- 
eral unrealistic features. For instance, in that example, the criterion of assembly 
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line output could be assessed reliably and accurately by counting the number of 
widgets produced by the workers. Not all criteria are that simple or easy to assess. 
Success in many occupational endeavors, such as management, medical practice, 
teaching, and so forth, may be judged on the bases of criteria that differ in terms 
of how teliably they can be assessed, how much control a worker has with respect 
to them, and the value the organization places on them. For example, the success 
of a manager may be gauged in terms of (a) worker productivity and (b) worker 
satisfaction, among other things. The skills and personal characteristics that make 
managers successful with regard to (a) are not necessarily the same as those lead- 
ing to success in (b), and in some situations these two criteria may even conflict 
with one another. Moreover, each of these criteria can be assessed by various 
methods. Productivity may be gauged by the quantity or the quality of produc- 
tion, or both; workers’ satisfaction may be measured through ratings of supervi- 
sors, by the amount of personnel turnover in a unit, and so forth. Test scores that 
can predict one facet of the criterion may have no correlation or may even be neg- 
atively correlated with those that predict another. 


Using Multiple Predictors 

The traditional way of dealing with the prediction of complex criteria, such as job 
performance, has been to use a test battery. In this context, the term battery refers 
to a combination of predictors especially selected to predict one or mote criteria. 
This meaning contrasts with the use of the term in clinical or counseling settings, 
where a battery usually refers to any group of tests that are administered to an in- 
dividual in the process of psychological assessment. The scores on the separate 
predictors in a test battery can be combined in a variety of ways, depending on the 
requirements of the selection or classification problem. Multiple regression tech- 
niques, fot instance, combine the scores on each test in the battery by inserting 
them into a linear regression equation that includes a numerical weight for each 
test score in the battery. Multiple regression equations are extensions of the simple lin- 
eat regression method presented in Table 5.5, but they involve multiple predic- 
tors instead of a single one. The weight for each predictor is directly proportional 
to its correlation with the criterion and inversely proportional to its correlation 
with the other predictors in the battery, so that test scores with the highest valid- 
ity and least amount of overlap with other scores are weighted most heavily. A 
multiple correlation coefficient (R) can then be computed to represent the cor- 
relation between the optimally weighted combination of test scores and the cri- 
terion. An alternative procedure for combining the scores in a test battery is 
through profile analysis. This method involves establishing a cutoff score for each 
predictor—based on its relation to the criterion—and results in the rejection of 
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all applicants whose scores fall below the minimum on any of the tests or, pos- 
sibly, on only those that assess skills that are considered critical for performing 
successfully on the criterion. 

Each of the two methods just described has some disadvantages. Multiple re- 
gression equations allow deficiencies in one ot more predictors to be compensated 
for by supetior performance in other predictors. The particular weighting of pre- 
dictors in these equations must also be checked on samples that are independent 
from the ones used to derive the equations, to see if the multiple correlation (R) 
holds up. Replication of predictor-criterion relationships on separate samples— 
which is a process known as cross-validation—is needed because any correlation co- 
efficient, regardless of its magnitude, is to some extent dependent on sample- 
specific error. Some reduction in the magnitude of the original R, or shrinkage, is 
expected upon cross-validation. To the extent that shrinkage is negligible, the orig- 
inal weights may be considered stable enough to be applied without further work. 

One of the major disadvantages of using the method of profile analysis, along 
with cutoff scores, is that this method typically fails to take into account the pos- 
sible unreliability of scores (see Quantifying Error in Test Scores: The Standard 
Error of Measurement, in Chapter 4). Another difficulty stems from the fact that 
having multiple cutoff scores may result in rejecting too many candidates, espe- 
cially those candidates from disadvantaged backgrounds who may score below 
the cutoff on one or more of the ability tests but might be able to overcome those 
deficiencies by virtue of training or a strong motivation to succeed. In general, the 
use of cutoff scores is justified only in situations when a deficit in a specific skill 
would have serious and deleterious consequences for job performance. One pos- 
sible solution, when this is the case, is to select on the basis of cutoff scores only 
for tests that assess the skills that are critical to the job and to use a regression 
equation for the other predictors in the battery. 


The Problem of Restricted Range in the Validation Samples 

As mentioned earlier, another unrealistic feature of the hypothetical validation 
study involving manual dexterity test scores presented earlier in this chapter is 
that it involved a heterogeneous sample of ten applicants for whom criterion 
measures did not yet exist at the time of testing. Most predictive validity studies 
do not proceed in this manner. Instead, they use samples of individuals for whom 
criterion data are already available, such as employees or students who have al- 
ready entered the jobs or educational pursuits for which the test under validation 
will be used. In other words, most of these studies use concurrent validation 
strategies to develop evidence of predictive validity. The kind of individuals for 
whom criterion measures are already available differ from those on whom the 
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test will eventually be used in that they have already been selected for employ- 
ment or admission into a program of study and have remained on the job or in 
school without being fired or dropping out. Because of this, we can almost always 
assume that their scores on the predictor tests undergoing validation, and on the 
criterion measures as well, will have a narrower range than would be the case with 
an unselected sample of applicants. It may be recalled from Chapter 2 that the ef- 
fect of a restriction in the range of either one of the variables is to reduce the size 
of the correlation coefficients. Thus, as a consequence of range restriction, the 
correlations between test scores and criteria (i.e., the validity coefficients) result- 
ing from these retrospective validation studies are usually smaller than would be 
the case if the samples were drawn from a more heterogeneous population, such 
as all those who apply for the jobs or academic programs in question. 


Validity Generalization 


The magnitude of the predictive validity indexes obtained for test scores depends 
on four basic elements: (a) the composition of the validation samples in terms of 
size and variability; (b) the nature and complexity of the criterion to be predicted; 
(c) the characteristics of the test itself; and (d) the interactions among all of these. 
Since each of these four factors can alter the results of validation studies, test 
users need to consider them carefully before assuming that the published evi- 
dence of test score validity derived from a single study will be applicable for their 
purposes and for their populations of test takers. 

With regard to the composition of validation samples, we have already discussed 
the problems attendant to criterion measures and range restriction. Small sample 
size frequently is also a problem. Most employers do not have large numbers of em- 
ployees in the same job category, and validation research findings based on small 
samples are mote prone to sample-specific error than those based on large 
samples. The bivariate data set for ten job applicants in our hypothetical example 
of predictive validity yielded an unrealistically large validity coefficient of .87. Al- 
though it is possible to obtain sizable correlations when individuals who participate 
in validation studies are not screened and criteria are narrowly construed, as in the 
example, the fact remains that local validation studies conducted using small 
samples, with a restricted range of scores and unreliable criteria, typically yield low 
and unstable estimates of the correlation between the predictor and criterion. 


Moderator Variables 
An additional issue related to the make-up of samples in predictive validation 
studies concerns the possible role of moderator variables. A moderator variable is 
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any characteristic of a subgroup of persons in a sample that influences the degree 
of correlation between two other variables. In theory, almost any demographic 
characteristic (e.g., sex, ethnicity, level of education, social class, geographic loca- 
tion, etc.) or psychological trait (interests, motivation, anxiety level, etc.) can act 
as a moderator variable in predictive validity studies and produce an interaction 
effect that either lowers or raises the predictor-criterion correlation. In order to 
pursue this possibility it is necessary to conduct separate validation studies or di- 
vide the validation samples into subgroups that differ on the variable that is pre- 
sumed to moderate validity coefficients. 

Considerable differences, in favor of Whites and Asians compared to Blacks 
ot Hispanics, have consistently been found in the average scores on tests of aca- 
demic abilities obtained by people from different racial or ethnic groups. These 
differences have engendered the suspicion that race or ethnicity may moderate 
the predictive validity of selection tests. As a result, many studies have conducted 
separate analyses of the magnitude of the predictor-criterion correlation and re- 
gression coefficients for Whites, Blacks, Hispanics, and members of other racial 
ot ethnic minorities. The purpose of such studies is to ascertain whether test 
scores have compatable validity for different groups and predict equally well for 
all, or whether test scores have different validities for different groups and are 
therefore biased. In this context, the term dias is used to indicate any systematic 
difference in the relationship between predictors and criteria for people belong- 
ing to different groups. Systematic differences can manifest themselves in two 
ways, namely, differential validity and differential prediction. 

Differential validity, in the context of test bias, refers to differences in the size of 
the correlations obtained between predictors and criteria for members of differ- 
ent groups. Differences in the magnitude of validity coefficients suggest that the 
test scores predict more accurately for members of the group with the larger co- 
efficient. Graphic evidence of differential validity is seen when the slopes of the 
regression lines for the two groups in question are different; the slope of the re- 
gression line is steeper for the group with the higher validity coefficient. Because 
of this, the problem of differential validity is also referred to as s/ope bias (see Table 
5.5 and Fig. 5.1). 

Differential prediction, on the other hand, occurs when test scores underpredict 
or overpredict the criterion performance of one group compared to the other. 
This problem is labeled żnzercept bias, because when a predictor overpredicts or un- 
derpredicts criterion performance for a group, the Y intercept, or point of origin 
of that group’s regression line on the Y axis, is different than for the other groups. 
With regard to the problems of differential validity and differential prediction for 
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test scores, several outcomes are possible, though not equally likely. Test scores 
may show (a) no bias with respect to different groups, (b) both differential valid- 
ity and differential prediction, (c) differential validity without differential predic- 
tion, or (d) differential prediction without differential validity. In general, the 
search for evidence that race acts as a moderator variable resulting in differential 
validity and differential prediction for members of racial minorities based on abil- 
ity test scores has not been very fruitful. In fact, studies that have investigated dif- 
ferences across ethnic groups in the accuracy of predictions of criterion perfor- 
mance indicate that tests often tend to overpredict the performance of Blacks and 
Hispanics, compated to Whites and Asians. On the other hand, some tests—es- 
pecially those used in educational admission decisions—sometimes underptre- 
dict the performance of women, albeit to a smaller extent than they overpredict 
the performance of some racial or ethnic minority groups (see, e.g., Young, 2001; 
Zwick, 2002, chaps. 5 & 6). 

Naturally, the reasons why some test scores generally overpredict criterion 
performance—typically in the form of grade point averages—for members of 
certain racial or ethnic minority groups and often underpredict the performance 
of women have been the subject of much conjecture and debate. The unreliabil- 
ity, and possible bias, of the criterion of grades is frequently cited as a possible ex- 
planation for the overprediction of the academic performance of racial or ethnic 
minorities, as are disparities in their upbringing or in the quality of their prior ed- 
ucational experiences. A novel explanation revolves around the notion of stereo- 
type threat, which refers to the deleterious effects that fear of confirming negative 
racial stereotypes seems to have on the test performance of some minority group 
members (Steele, 1997). As far as the underprediction of the college performance 
of women, the most frequently cited conjectures center on the fact that, as op- 
posed to men, women as a group (a) tend to choose courses that are less strin- 
gently graded or (b) are more serious about their studies. Although it may be true 
that these and other variables related to ethnic and gender status influence the test 
scores and criterion performance of different groups, as well as the correlations 
between them, it is not always possible to establish precisely what these factors 
are. Moreover, it appears that the extent of differential prediction of the college 
performance for racial or ethnic minority groups and women has been decreas- 
ing over the past quarter century (Young, 2001). Furthermore, whatever these 
vatiables are, they obviously do not apply to all members of those groups— 
which are themselves quite heterogeneous—in the same fashion. Nevertheless, 
the possibility of differential validity for members of different ethnic groups, sex 
groups, non-native English speakers, and other traditionally disadvantaged cate- 
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gories of individuals always needs to be investigated to ascertain that test scores 
used in high-stakes decisions are fair to all. 

One possible solution to the problem of differential prediction of test scores 
would be to use different regression equations, and different cutoff scores, for se- 
lection of individuals from different ethnic groups and genders. In the case of test 
scores that overpredict the performance of Blacks and Hispanics, for instance, 
this would mean requiring higher cutoff scores for members of those minorities 
than for Whites and Asians. However, this obviously runs counter to the goals of 
extending opportunity to groups through affirmative action and of increasing di- 
versity in higher education and in the workplace. Another proposed solution, im- 
plemented in the 1980s with the General Aptitude Test Battery (GATB) devel- 
oped by the United States Employment Service (USES), is to use subgroup 
norms to ensure comparable rates of employment referrals for Blacks, Hispanics, 
and Whites. This practice generated so much opposition that it led to the passage 
of the Civil Rights Act of 1991 (P.L. 101-336), which banned any kind of score 
adjustment based on race, color, sex, religion, or national origin. In light of these 
obstacles, most of the work in the area of fairness in testing is now concentrated 
on (a) identifying factors that may be contributing to differential prediction 
across gender and ethnic groups (see, e.g., Steele, 1997; Willingham, Pollack, & 
Lewis, 2000) and (b) analyzing how tests items function for different subgroups, 
while tests ate under construction, to make sure that those which function dif- 
ferently for different subgroups are not included (see Chapter 6). 


Meta-analyses 

Since the late 1970s, a good deal of clarity and renewed enthusiasm has been 
brought to beat on the somewhat pes- 
simistic outlook stemming from ear- 


DON’T FORGET lier research on the validity of selec- 
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rely on a series of quantitative procedures that provide for the synthesis and inte- 
gration of the results obtained from the research literature on a given subject. 
These techniques, which had been used in other scientific fields for some time, 
were introduced into psychometric research by Schmidt and Hunter (1977) as a 
way to approach the problem of validity generalization. In the past couple of 
decades, meta-analytic techniques have actually demonstrated that the predictive 
validity of test scores is not as situationally specific as previously thought and 
have become an important method for clarifying conflicting findings in other 
portions of the psychological literature (Hunter & Schmidt, 1990, 1996). 
Widespread interest in, and use of, meta-analyses has been stimulated by the 
realization that many conflicting findings in psychological research—including 
those of validation studies—are attributable to the imperfections of individual 
studies. When the influence of artifacts such as sampling error, measurement er- 
ror, range restriction, and unjustified dichotomization of variables are removed 
through statistical corrections, a much clearer picture emerges. Concomitantly, 
there has been a growing recognition that hypothesis testing in psychological 
studies has overemphasized statistical significance levels that stress the avoidance of 
Type I errors (i.e., incorrectly rejecting the null hypothesis of no difference when it 
is true) while neglecting the possibility of Type I errors (i.e., incorrectly accepting 
the null hypothesis when it is false). Since the relation between Type I and Type 
II errors is inverse, the emphasis on avoiding Type I increases the likelihood of 
Type I errors. As a consequence, an enormous number of research results that 
do not reach the desired levels of statistical significance (e.g., .05 or .01), but nev- 
ertheless can contribute valuable information, have been either ignored or left 
out of the literature. Rapid Reference 5.10 lists some references that provide ad- 
ditional information on the disadvantages and advantages of null hypothesis sig- 
nificance tests. At any rate, these discussions have led to what most investigators 
see as a salutary change in the way that research findings are reported. Instead of 
merely stating the significance levels or probability of results, it is now considered 
necessary to include indexes of the effect sizes, or the strength of the relationships 
found bya research study, along with confidence intervals for the effect sizes and 
for all estimates of parameters resulting from an investigation (APA, 2001). 
Although the methodology of meta-analyses is still evolving, it has already 
made substantial contributions to the evidence of the predictive validity of test 
scores and non-test procedures—such as employment interviews and biograph- 
ical data inventories—that are used in personnel selection (Hartigan & Wigdor, 
1989; Schmidt & Hunter, 1998; Schmidt et al., 1993). In addition, meta-analyses 
have helped to clarify the research literature and to further the development of 
theories in several areas of industrial-organizational psychology—such as the re- 
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= fad Reference 5.10 


Sources of Information on the Pros and Cons of 
Significance Tests 


A debate about the merits and drawbacks inherent in the use of null hypothesis 
tests of statistical significance for making inferences in social science research has 
been going on for decades among statisticians and methodologists. Some of them 
have been so convinced that these tests have a detrimental effect on the scientific 
enterprise that they have suggested a ban of significance testing in research re- 
ports. Although no such ban has been instituted, it is now common for psycho- 
logical journals, as well as journals in most other related disciplines, to require 
effect-size estimates whenever probability (p) values are reported, along with 
confidence intervals for effect sizes, correlation coefficients, and other estimates 
of population parameters. Additional information about the issues involved in this 
debate can be found in the following sources: 


* Abelson, R. P (1997). On the surprising longevity of flogged horses: Why there 

is a case for the significance test. Psychological Science, 8, 12-15. 

* Cohen, J. (1994). The earth is round (p < .05).American Psychologist, 49, 
997-1003. 

e Thompson, B. (2002). What future quantitative social science research could 
look like: Confidence intervals for effect sizes. Educational Researcher, 3 1, 25-32. 

e Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical meth- 


ods in psychology journals: Guidelines and explanations. American Psychologist, 
54, 594-604. 


lation between job satisfaction and job performance—as well as in various other 
fields (see, e.g., Hunter & Schmidt, 1996; Kirsch & Sapirstein, 1998; Rosenthal & 
DiMatteo, 2001). An example from the area of admissions testing in higher edu- 
cation will illustrate the potential inherent in meta-analytic research. 

An educational case in point. Studies of the validity of Graduate Record Examination 
(GRE) scores as predictors of performance in graduate school programs have a 
long history—dating back to the 1940s—that has been plagued by inconsistent 
findings. Whereas some investigations (e.g, Briel, O’Neill, & Scheuneman, 1993; 
Broadus & Elmore, 1983) found the GRE General and Subject tests to be fairly 
valid predictors of graduate school performance, many others—including some 
limited meta-analytic studies—concluded that the relationship between GRE 
scores and various indexes of success in graduate school was less than adequate 
(e.g, Goldberg & Alliger, 1992; Marston, 1971; Morrison & Morrison, 1995; 
Sternberg & Williams, 1997). Many GRE validity studies yielded coefficients 
ranging from small negative correlations to positive correlations in the low .20s 
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for the Verbal and Quantitative scores of the GRE, and somewhat higher coeffi- 
cients for the GRE Subject test scores. Although some of these findings were crit- 
icized on the basis of methodological artifacts such as highly restricted ranges in 
both GRE scores and criterion measures, as well as unreliability of criteria, the 
general tenor of the literature on the validity of GRE scores did not seem to pro- 
vide substantial evidence to support their use in graduate school admission deci- 
sions (see, e.g., Kuncel, Campbell, & Ones, 1998). 

Against this background, Kuncel, Hezlett, and Ones (2001) recently con- 
ducted a meticulous and comprehensive meta-analysis of the GRE data from 
1,753 independent samples comprising a total of 82,659 graduate students. This 
study systematically addressed theoretical, statistical, and methodological aspects 
of the literature on the predictive validity of GRE scores. Kuncel and his col- 
leagues examined the relationships between five predictors—GRE Verbal 
(GRE-V), Quantitative (GRE-Q), Analytical (GRE-A), and Subject test scores as 
well as undergraduate grade point average (UGPA)—and eight different criteria 
of graduate school success, including first-year GPA and graduate GPA (GGPA), 
comprehensive examination scores, faculty ratings, degree attainment, and nu- 
merical indexes related to research productivity. They conducted separate analy- 
ses for the total sample and for subsamples representing students in the areas of 
humanities, social sciences, life sciences, and math-physical sciences, as well as for 
non-native English speakers and for students older than the traditional graduate 
school age. Among the many methodological refinements Kuncel and colleagues 
employed in their analyses were corrections for range restriction and for changes 
in the variability of distributions of both predictor and criterion variables as well 
as for the unreliability of criterion measures. In addition, these investigators ad- 
dressed a number of potential pitfalls inherent in meta-analyses. 

Kuncel and his colleagues (2001) report their major results in terms of average 
observed correlations, along with their standard deviations, weighted for sample 
size. They also report estimated operational validities with their standard devia- 
tions and 90% confidence intervals. Those findings indicate that the four GRE 
measures (GRE-V, GRE-Q, GRE-A, and GRE Subject tests) are reasonably 
good predictors of most of the criteria employed for the total sample as well as 
for the subsamples. In fact, in most cases GRE scores seem to be better predic- 
tors than UGPA. The GRE Subject test scores turned out to be the best single 
predictors of graduate GPA across all disciplines, with estimated operational va- 
lidities ranging from .40 to .49. In contrast, the scores on the general tests 
(GRE-V, GRE-Q, and GRE-A)—while correlating substantially with those of 
the Subject tests—had operational validity coefficients ranging between .27 and 
.48. Kuncel and colleagues conclude that although the GRE general test scores 
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contribute only a small increment in validity when added to the Subject test 
scores, they can still be of value, especially for students whose undergraduate de- 
grees are in areas other than the ones in which they are applying for admission. 

In short, Kuncel and colleagues’ (2001) meta-analysis clearly suggests (a) that 
much of the inconsistency in previous GRE validation studies was the result of 
range restriction and sampling error in those studies and (b) that GRE scores de- 
serve a role in the process of graduate school admissions. However, these authors 
did not investigate the issue of differential prediction for women and members of 
racial or ethnic minorities, and they do concede that there is still much room for 
improving the validity of the graduate school admissions process. With regard to 
the latter point, the following notions should be kept in mind: 


* The purpose of GRE scores and most other predictors used in selec- 
tion decisions is not to estimate the exact criterion standing of appli- 
cants but rather to determine whether applicants can achieve the neces- 
sary level of success. If criterion scores had to be predicted exactly, the 
margin of error (SE...) for coefficients in the .30s and .40s would indeed 
be considerable. 

e Performance on most criteria, including graduate school success, is de- 
termined by multiple factors, including attitudinal and emotional char- 
acteristics and behavioral habits, as well as creative and practical talents 
that ate not measured by the GRE or by other cognitive tests ordinarily 
used as predictors. 

e Most other selection decisions, including graduate school admissions, 
are rarely, if ever, made solely on the basis of a single predictor. There- 

fore, the crucial issue with regard 
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efficiency in decision-making (see, e.g., Hartigan & Wigdor, 1989; Ko- 
brin, Camara, & Milewski, 2002; Kuncel, Hezlett, & Ones, 2001). 


Beyond Selection: Using Test Scores for Other Types of Decisions 


Up to this point the discussion of criterion-related validation procedures has cen- 
tered primarily on the use of tests for selection or screening made on the bases of 
either concurrent or predictive test score validity. Se/ection decisions are those that 
require a choice between two alternatives. In employment and educational set- 
tings, the usual alternatives are whether to accept or reject an applicant; in clini- 
cal and forensic settings, selection decisions usually involve a determination of 
whether a particular syndrome or condition is present or absent. The term screen- 
ing refers to a preliminary step of a selection process usually undertaken to sepa- 
rate individuals who merit or require more extensive evaluation from those who 
do not. For example, many clinics periodically use a simple and short question- 
naire to screen for disorders such as depression or anxiety in the general popula- 
tion; employers may screen applicants with a brief instrument in order to limit the 
pool of applicants to those who meet the minimal requirements for a job. 

Psychological test scores are also used for making placement and classification 
decisions, both of which involve more than two options. Of these two, placement 
decisions are simpler. They involve assigning individuals to separate categories or 
treatments on the basis of a single score, or of a composite score computed from 
a single regression equation, with reference to a single criterion. Although place- 
ment decisions do not involve the option of rejecting individuals who do not 
meet a certain level of performance on a test or predictor, they are not substan- 
tially different from selection decisions in terms of the evidence they require, 
which is a demonstrable relationship between one or more predictors and a cri- 
terion. Scores on a reading test, for instance, may be used to place students in in- 
structional groups suitable to their levels of reading skills. Similarly, scores on a 
depression scale might be used to classify psychiatric patients in terms of the 
severity of their depressive symptoms to help determine appropriate types and 
levels of therapeutic intervention. 

Classification decisions, on the other hand, are a good deal more complicated. 
In classification—as in placement—nobody is rejected, but individuals must be 
differentially assigned to distinct categories or treatments on the bases of multiple 
criteria. This means that multiple predictors are required and their relationships 
with each criterion have to be determined independently, through separate re- 
gression equations. The most appropriate tool for classification decisions is a bat- 
tery of tests or predictors whose results are validated against the various criteria 


204 ESSENTIALS OF PSYCHOLOGICAL TESTING > 


to be predicted and then combined in equations that reflect their relative weights 
for the prediction of each criterion. 

Classification decisions are required in employment, educational, counseling, 
and clinical settings. In the realm of employment, including military or industrial 
settings, these decisions are necessary when the aptitudes of an available person- 
nel pool have to be evaluated in order to assign individuals to the jobs or training 
programs in which they are most likely to function effectively. Vocational coun- 
seling of individuals wanting to decide ona program of study or career choice also 
calls for classifications decisions. In clinical settings, classification decisions must 
be made in cases that require differential diagnosis. A typical example would be 
the need to establish whether an older patient who shows symptoms of depres- 
sion and memory problems may be suffering from a depressive disorder that af- 
fects memory and concentration, from an incipient dementing process that is 
causing the depression, or from a combination of the two. 

Batteries of tests that are used for classification decisions must be evaluated on 
evidence of differential validity. Within this context, the term differential validity 
means that a battery should be able to predict, or establish, differences among 
two or more criteria. In a two-criterion classification problem an ideal battery 
would consist of predictors that correlate highly with one criterion and not at all, 
ot negatively, with the other. In the problem of differential diagnosis of depres- 
sion versus dementia, for instance, one might look for differences in the tempo- 
tal sequencing of symptoms of depression and cognitive impairment or for dif- 
ferences in relative levels of performance on various types of memory tests. 

When the classification situation involves predictions against more than two 
criteria, such as assigning personnel to any one of several possible jobs or train- 
ing programs, the problem of establishing validity evidence becomes even more 
complex. In this kind of condition, a predictor that correlates equally well with all 
of the criteria involved in the decision—such as a test of general intelligence with 
respect to most job performance criteria—is of relatively little use. One possible 
way of handling classification problems of this type is through the use of mul- 
tiple discriminant function analyses. Discriminant functions involve the application of 
weighted combinations of scores on the predictors—derived by means of re- 
gression analyses—to determine how closely an individual’s profile of scores 
matches the profiles of individuals in different occupational groups, different 
specialties, or different psychiatric categories. Although discriminant functions 
are useful in certain instances (e.g., when criteria consist simply of membership 
in one group versus another or when there is a nonlinear relationship between a 
criterion and one or more of the predictors), they fall short in terms of the re- 
quirements of many situations because they do not allow for the prediction of 
level of success in a specific field. For an example of the application of discrimi- 
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nant function analysis in the differentiation of WAIS-R profiles of nonlitigating 
head-injured patients from those of subjects instructed to feign head trauma, see 
Mittenberg, Theroux-Fichera, Zielinski, and Heilbronner (1995). 

Another traditional strategy that can be used for both selection and classifica- 
tion problems is synthetic validation (Balma, 1959). This technique essentially relies 
on detailed job analyses that identify specific job components and their relative 
weights in different jobs. Based on such analyses, previously established regres- 
sion coefficients for test scores that predict those separate job elements can be 
combined into a new, synthetic battery that will predict performance on the jobs 
in question. Statistical procedures associated with this method were developed by 
Primoff (1959; Primoff & Eyde, 1988) and have been expanded by others since 
then. However, in order to be useful in classification decisions, synthetic valida- 
tion strategies must involve predictors that show good discriminant validity, un- 
less the criterion components themselves are correlated substantially (Guion, 
1998, pp. 354-355). 


Additional Perspectives on Criterion-Related Validation 


Several of the methodological advances discussed earlier in this chapter, such as 
structural equation modeling and meta-analytic validity generalization studies, 
have been brought to bear on recent research on the problems of criterion- 
related validation. At the same time, the availability of these increasingly sophis- 
ticated statistical tools has focused attention on the conceptualization of both 
predictors and performance criteria, as well as their interrelationships (see, e.g., 
J. P. Campbell, 1990). One of the most significant recent developments, in terms 
of predictors, is the recognition that the inclusion of factors related to personal- 
ity dimensions—in addition to those related to abilities—can increase the valid- 
ity of instruments used to predict performance in a number of arenas (see, e.g., 
Lubinski & Dawis, 1992). With regard to the criterion problem, the multifaceted 
nature of performance in most jobs and educational endeavors is now widely ac- 
knowledged and increasingly scrutinized. This requites analyzing the various el- 
ements that make for success, assessing their relative value, recognizing which 
ones are under the control of the individual, developing methods to evaluate each 
element separately, and then combining them into a total measure of perfor- 
mance that takes into account all of these factors and their relative weights. 


A Model Program of Criterion-Related Validation 


An outstanding example of the application of many innovations in validation re- 
search methods can be found in the work that John P. Campbell and his col- 
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leagues (J. P. Campbell, 1990, 1994; J. P. Campbell & Knapp, 2001) have con- 
ducted over the past two decades, in conjunction with a comprehensive effort to 
evaluate and improve the U.S. Army’s procedures for personnel selection and 
classification, known as Project A. A complete account of this work, which may 
possibly be the largest single project in the history of personnel research, is not 
feasible in this context. However, some of its highlights are presented here to give 
readers an idea of its scope and significance. 


Project A 

Using an extensive database of more than 50,000 people, the Project A investiga- 
tors selected 21 military occupational specialties (MOSs), out of a total of more 
than 200 Army entry-level jobs, on which to conduct both concurrent and longi- 
tudinal predictive validation studies. The predictors studied included the ten sub- 
tests of the Armed Services Vocational Aptitude Battery (ASVAB) listed in Rapid 
Reference 5.11. Various composite scores consisting of different subtest combi- 
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Armed Services Vocational Aptitude Battery (ASVAB) 


The ASVAB is administered as a computerized adaptive test (CAT-ASVAB) at mil- 
itary entrance processing stations and as a paper-and-pencil test at mobile test 
sites. Individuals who take the ASVAB have already been pretested with shorter 
screening instruments administered by recruiters. The ten subtests that make up 
the ASVAB, used by all the U.S. Armed Forces for selection and classification of 
personnel, are the following: 


e Arithmetic Reasoning 

e Numerical Operations 

e Paragraph Comprehension 
e Word Knowledge 

* Coding Speed 


* General Science 

e Mathematics Knowledge 

e Electronics Information 

e Mechanical Comprehension 

e Automotive-Shop Information 


The first four of these subtests make up the Armed Forces Qualification Test 
(AFQT) composite, which, along with high school graduation status, is used to 
determine eligibility for enlistment. The ASVAB subtests are also combined into 
ten different aptitude composite scores that are used as predictors of success at 
the Army school training programs (J.P Campbell & Knapp, 2001, pp. 14-18). 
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nations from the ASVAB have traditionally been used for selection and classifi- 
cation of individuals who apply for service in the Armed Forces. In addition, sev- 
eral new instruments—including tests of psychomotor and spatial abilities as 
well as personality and interest measures—were also included in the validation 
work. 

Project A investigators carried out extensive job analyses and paid special at- 
tention to the standardization of measures of job proficiency for each of the 
MOSs. Exploratory and confirmatory factor analyses, as well as other statistical 
techniques, were employed in performance modeling studies that led to the spec- 
ification of the factor scores that were used as criteria at three different career 
stages: end of training performance, first-tour job performance, and second-tour 
job performance. Exploration of validity evidence at various career stages was a 
significant aspect of this research because, in addition to developing a test battery 
capable of differential prediction, another important goal of the personnel man- 
agement task in the All Volunteer Armed Fotces is to minimize attrition. 

The actual validation analyses performed through the duration of Project A in- 
cluded (a) correlations between each predictor and performance criteria; (b) 
comparisons of the incremental validity provided by the experimental measures 
with respect to each criterion, over and above the predictive power of the previ- 
ously available ASVAB scores; (c) development and comparison of various opti- 
mal equations for maximum validity in the longitudinal prediction of first-tour 
performance; and (d) analyses of the validity of alternative equations using dif- 
ferent combinations of test data and previous performance for the prediction of 
second-tour performance. Validity generalization and synthetic validation meth- 
ods were used to investigate the efficiency of various combinations of predictors 
for the 21 MOSs originally selected for analysis as well as for many other MOSs 
and clusters of occupations. Additional studies included an evaluation of the util- 
ity and gains achieved through alternative classification strategies and conditions. 

Among the many useful findings of Project A, one of the most significant has 
been the corroboration of the potential value of several of the new measures that 
were studied, including those related to personality dimensions, as well as some 
new tests of psychomotor and spatial abilities. These additional predictors are 
now at various preliminary stages of implementation and are being explored fur- 
ther. According to the principal Project A investigators, these measures are not 
yet fully operational due to obstacles posed by the many parties involved in the 
Army selection and classification system as well as by organizational inertia (J. P. 
Campbell & Knapp, 2001, pp. 570-574). Nevertheless, Project A and the subse- 
quent investigations it has spawned have already contributed a number of sub- 
stantive and methodological advances that are sure to improve the caliber of val- 
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DON’T FORGET 


The judgment concerning the validity of test scores is relative. When the evidence 
that has accumulated concerning the validity of scores produced by a test is con- 
sidered in the abstract, it may or may not be deemed sufficient for the test's in- 
tended purpose. 


However, when any specific score or set of scores from an individual or group is 
considered, it must be understood that the scores may have been affected by fac- 
tors uniquely pertinent to the test takers, the examiner, the context in which test- 
ing takes place, and the interaction between these factors. Thus, the testing situa- 
tion must always be taken into account when test scores are interpreted. 


In addition, making inferences on the bases of test scores requires information 
about their frames of reference and their reliability, as well as validity evidence 
from all pertinent sources. 


idation research for selection and classification, both in the military and in many 
other contexts, and thus will help in the overarching goal of maximizing the uti- 
lization of human talents. 


ADDITIONAL ASPECTS OF VALIDITY: 
UTILITY AND CONSEQUENCES 


There are two significant aspects concerning the use of test scores that are closely 
connected to their validity but are not necessarily of its essence, namely, their util- 
ity and the consequences attendant to their uses. The complexity of these topics 
prevents an extended discussion of them within the scope of this volume. How- 
ever, their crucial importance to the psychological testing enterprise warrants 
their introduction at this point, to be followed by a more extensive treatment in 
Chapter 7. 


Evaluating the Utility of Testing 


The zzility of tests, and test scores, refers to the benefits they bring to decision- 
making. Utility is contingent on the extent to which the use of tests can increase 
the rate of accuracy of the inferences and decisions we wish to make—over and 
above what it would be if we used other available tools. Typically, utility is assessed 
in economic terms, such as the cost-benefit ratios involved in using tests versus 
non-test data. Given that test use always takes place within a context, the analysis 
of its costs and benefits necessarily must take into account additional data perti- 
nent to each particular situation in which the use of tests is contemplated. De- 
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pending on the context, these data include matters such as the probabilities and 
risks involved in making false positive and false negative determinations, the 
availability of alternative tools—and their relative efficiency compared to the 
tests—as well as the relative ease or difficulty of the determinations that need to 
be made. This aspect of test use is part of the much larger topic of decision the- 
ory, which applies not only to psychometrics, in particular, and psychology, in 
general, but also to such diverse fields as medicine, economics, jurisprudence, 
military science, and gaming, as well as any other human endeavor in which 
strategic planning is required. For a sample of the numerous contributions psy- 
chometric experts have made to decision theory over the past few decades, see 
Boudreau (1991), Brogden (1946), Brown and Ghiselli (1953), Buchwald (1965), 
Cronbach and Gleser (1965), Hunter and Schmidt (1981), Schmidt, Hunter, 
McKenzie, & Muldrow (1979) and Taylor and Russell (1939). An excellent dis- 
cussion of the statistical aspects of decision theory within the context of enhanc- 
ing the accuracy and utility of diagnostic decisions is available in a recent report 
by Swets, Dawes, and Monahan (2000). 


Assessing the Consequences of Test Use 


The individual and social consequences of test use, which can be either positive 
or negative, have to be assessed in terms of their value implications. Some theo- 
tists, notably Messick (1989, 1995), have argued that validity judgments are actu- 
ally value judgments. Messick has proposed the inclusion of consequential aspects 
of the use and interpretation of test scores—that is, the evaluation of their in- 
tended and unintended social consequences—within the notion of validity in its 
most comprehensive sense. In criticizing Messick’s proposal some have labeled it 
“consequential validity,’ a term that Messick himself never used. Be that as it may, 
this particular aspect of Messick’s immense contributions to psychometric theory 
and practice has not been widely adopted by the testing profession. Most would 
argue that while the consequential aspects of test use are of paramount impor- 
tance, and should be investigated prior to implementation and documented af- 
terward, they lie within the realms of professional ethics, moral values, and polit- 
ical considerations rather than of validity determination as such (Cole & Moss, 
1989; Lees-Haley, 1996; Linn, 1998). 

The Ethical Principles of Psychologists and Code of Conduct (APA, 2002), for in- 
stance, enjoins all psychologists—including those who use tests and other as- 
sessment tools—to take into account the possible ramifications of their work in 
order to maximize benefits and prevent or minimize harm to others. Users of 
tests and assessment tools, specifically, are bound to guard against misinterpreta- 
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tions and misuses and to obtain the consent of test takers, prior to testing, as to 
the purposes of testing, the manner in which scores will be used, the persons to 
whom scores will be released, and other such matters. Similar constraints apply 
to the work of test authors and developers, test publishers, and test reviewers, as 
well as other professionals involved in the testing enterprise (e.g., Society for In- 
dustrial and Organizational Psychology [SIOP], 2003). To add to the already 
complex notion of test score validation the additional facet of evaluating the ram- 
ifications of test use in terms of broader social concerns—such as balancing 
moral principles like justice and fairness—would place an undue burden, which 
belongs to society as a whole, on the testing profession. 

An example of how the ethical principles and practices in testing may be ap- 
plied to the realm of education can be found ina resource guide for educators and 
policymakers that presents both the professional standards and the legal prin- 
ciples pertinent to the use of tests in making high-stakes educational decisions for 
students (U.S. Department of Education, Office for Civil Rights, 2000). Similar 
sets of guidelines for the use of tests in other specialty fields will be discussed in 
Chapter 7 (e.g., SIOP, 2003). 


CONCLUDING COMMENTS 


Rapid Reference 5.12 outlines how various sources of evidence and validation 
strategies can be brought to bear in the interpretation of scores from a single test, 
depending on the purposes for which the test is used. In most cases, to the extent 
that the proposed interpretation of the scores on a test moves away from the orig- 
inal purpose for which the test was developed, the lines of evidence for alterna- 
tive uses and interpretations will become less direct. For instance, the test used as 
an example in Rapid Reference 5.12 was a final exam in a Calculus I course; thus, 
we can infer that its original purpose was to determine whether test takers had 
mastered enough of the content in that course to achieve a passing grade. As the 
interpretation of the scores on this test is extended to determining readiness for 
Calculus II, predicting success as a mathematics major, or using test scores as a 
proxy for mathematical ability in an investigation of correlates of personality 
type, the link between the validation evidence and the intended interpretation be- 
comes more and mote tenuous. This does not mean that the final exam in Calcu- 
lus I should not be used for purposes other than the original one, but it does sug- 
gest that additional evidence will be required for those other purposes. 

In the present chapter we have discussed the evidence for validity of test scores 
from the point of view of the test as whole. In the next, we turn to a more minute 
analysis of test data from the perspective of the behavior sample units that make 
up test scores, namely, test items. 
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‘a TEST YOURSELF wA. 


|. Validity is the degree to which 


a) atest measures what it purports to measure. 
b) evidence supports inferences from test scores. 
c) test scores are consistent across situations. 


2. In recent decades the various forms of validity evidence have been sub- 
sumed within the notion of validity. 


a) content 


b) concurrent 


c) predictive 


d) construct 

3. Nomothetic span refers to 

a) anetwork of relationships between measures. 
b) the decomposition of tasks. 

c) identifying differences among test takers. 

d) the scope of the construct being measured. 


4. Evidence of validity that is based on test content and response processes 
is particularly applicable to 


a) interest inventories. 
b) educational tests. 


AO 


personality tests. 

5. Face validity refers primarily to 

a) the representativeness of test content. 

b) evidence of validity from the psychometric perspective. 


c) the superficial characteristics of a test. 
d) the amount of empirical validation data accumulated for a test. 


6. In order to gather discriminant validity evidence, one would correlate 
the scores of tests that purport to assess constructs. 


a) the same 
b) similar, but not the same 
c) different 


7. One of the most useful aspects of factor analysis, as applied to test valida- 
tion research, is that the results of applying this technique can 


a 


n 


implify the interpretation and reporting of test scores. 


b) reveal the essential aspects of the constructs that tests are assessing. 


O 


readily be generalized across populations. 
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8. Which of the following statements about criterion measures is not true? 
a) Criterion measures can differ in terms of their reliability and validity. 

b) Different criterion measures do not always correlate with each other. 

c) Criterion measures may or may not generalize across different groups. 
d) The best criterion measures are usually available at the time of testing. 


9. Standard errors of estimate are used in order to gauge the 


a) reliability of criteria. 

b) reliability of predictors. 

c) accuracy of obtained scores. 

d) accuracy with which criteria are predicted. 


10. From the standpoint of criterion-related validation procedures, which of 
the following types of decisions is the most complex? 


a) Selection 
b) Placement 
c) Classification 


Answers: |.b; 2. d; Sra Aoo ee a8 a rO: 


Six 


ESSENTIAL TEST ITEM CONSIDERATIONS 


est items are the units that make up a test and the means through which 

samples of test takers’ behavior are gathered. It follows that the overall 

quality of a test depends primarily on the quality of the items that make it 
up, although the number of items in a test, and theit sequencing or position 
within the test, are also matters of fundamental importance. Just as tests are eval- 
uated with regard to the extent to which they meet their intended purposes, indi- 
vidual items must be evaluated based on the extent to which they meet the pur- 
poses of the test as a whole. [tem analysis is a general term that refers to all the 
techniques used to assess the characteristics of test items and evaluate their qual- 
ity during the process of test development and test construction. 

Item analysis involves both qualitative and quantitative procedures. Qualitative 
item analysis procedures tely on the judgments of reviewers concerning the sub- 
stantive and stylistic characteristics of items as well as their accuracy and fairness. 
The major criteria used to evaluate items qualitatively are (a) appropriateness of 
item content and format to the purpose of the test and the populations for whom 
the test is designed, (b) clarity of expression, (c) grammatical correctness, and (d) 
adherence to some basic rules for writing items that have evolved over time. As 
discussed later in this chapter, item content is also carefully examined to identify 
and eliminate possible sources of bias or offensive portrayals of any specific sub- 
group of the population. Rapid Reference 6.1 lists books that provide informa- 
tion about the process of item development and practical guidelines for writing 
test items. Quantitative item analysis involves a variety of statistical procedures de- 
signed to ascertain the psychometric characteristics of items based on the re- 
sponses obtained from the samples used in the process of test development. 
Most of the remainder of this chapter deals with the quantitative analysis of test 
items. 
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= faid Reference 6./ 


Writing Test Items 


For insight into the process of preparing items for ability tests, as well as explicit 
guidance on how to write them, readers may wish to consult one of the following 
sources: 


* Bennett, R. E., & Ward, W. C. (Eds.). (1993). Construction versus choice: Issues in 
constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: 

Erlbaum. 

Haladyna, T. M. (1997). Writing test items to evaluate higher order thinking. Boston: 

Allyn & Bacon. 

e Haladyna, T. M. (1999). Developing and validating multiple-choice test items (2nd 
ed.). Mahwah, NJ: Erlbaum. 

Although no comparable guidebooks exist for all of the vast range of approaches 

to the development of personality assessment instruments, some basic principles 

for preparing objective items can be gleaned from the following books: 


e Aiken, L. R. (1996). Rating scales and checklists: Evaluating behavior, personality 
and attitudes. New York: Wiley. 


e Aiken, L. R. (1997). Questionnaires and inventories: Surveying opinions and assess- 
ing personality. New York: Wiley. 


e Fink, A. (2002). How to ask survey questions (2nd ed., Vol. 2). Thousand Oaks, 
CA: Sage. [ This is one of the ten volumes in the Survey Kit, edited by Arlene 
Fink and published by Sage.] 


THE CONTEXT OF ITEM ANALYSIS: TEST DEVELOPMENT 


Fundamentally, the procedures involved in item generation, item selection, and 
item analysis pertain to the topic of test theory and design. As such, they are of 
critical importance to test authors and test developers. Test users need to be fa- 
miliar with these procedures in order to understand the nature of the tasks in- 
volved in a test and evaluate the instruments they select. However, by the time a 
test is made available to test users, it is already a finished product. From the point 
of view of test usets, once a test has been selected, its items are of interest—es- 
pecially in the context of individual assessment—primarily as a means of ob- 
serving and inspecting test takers’ responses from the unique perspective of the 
specific situation and circumstances in which a test is administered. In individual 
assessment, the ways in which examinees respond to test tasks and their particu- 
lar response patterns can provide additional information to supplement the pro- 
cess of test score interpretation. Test users, naturally, are also concerned with the 
practical features of test items. Chief among these features are the appropriate- 
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ness of items for specific types of settings and examinees, the ease with which 
items can be administered and scored, the time involved in administering the 
items, and the amount of training required to master the procedures involved in 
the administration and scoring of items. 

Item analysis procedures are implemented at various points during the process 
of test development, a process that includes several other steps. In order to pro- 
vide a context for the discussion of item analysis, the steps involved in develop- 
ing a test are described briefly in the next few paragraphs. More extensive treat- 
ments of the test development process are available in many sources (e.g., AERA, 
APA, & NCME, 1999, chap. 3; DeVellis, 2003; Ramsay & Reynolds, 2000; 
Robertson, 1992). 

As Robertson (1992) makes clear, developing a standardized test entails a con- 
siderable investment of time and money and requires specialized professional ex- 
pertise in psychometrics as well as in the particular area with which the test deals. 
Because of this, the development of tests intended for commercial distribution— 
as opposed to experimental measures to be used primarily for research purposes, 
classroom tests, or tests developed by employers for in-house use—is typically 
undertaken by test publishing firms that have the necessary financial resources 
and technical expertise. The impetus for new tests may stem either from the test 
publishers’ own staffs or from independent test authors and investigators who 
submit their ideas to publishers. 

The decision to develop a test usually is made when the prospective test devel- 
oper realizes either that no test exists for a particular purpose or that the existing 
tests for a certain purpose are not adequate for one reason or another. Marketing 
considerations are also central to the decision-making process in commercial test 
publishing. At any rate, as the decision to develop a test is made, and its purpose 
and rationale are carefully articulated in terms of the sorts of inferences to be 
drawn from test scores, the test developer must also make a plan for the test. 

Planning a test entails specifying (a) the constructs or knowledge domains that 
the test will assess, (b) the type of population with which the test will be used, (c) 
the objectives of the items to be developed, within the framework of the test’s 
purpose, and (d) the concrete means through which the behavior samples will be 
gathered and scored. This last point includes decisions about the method of ad- 
ministration, the format of test item stimuli and responses, and the scoring pro- 
cedures to be used. After these issues are decided and a preliminary plan for the 
test is made, the process of test development usually involves the following steps: 


1. Generating the item pool by writing or otherwise creating the test 
items, as well as the administration and scoring procedures to be used; 
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. Submitting the item pool to reviewers for qualitative item analysis, and 
revising or replacing items as needed; 

. Trying out the items that have been generated and reviewed on 
samples that are representative of the population for whom the test is 
intended; 

. Evaluating the results of trial administrations of the item pool through 
quantitative item analysis and additional qualitative analysis; 

. Adding, deleting, and/or modifying items as needed, on the basis of 
both qualitative and quantitative item analyses; 

. Conducting additional trial administrations for the purpose of check- 
ing whether item statistics remain stable across different groups—a 
process known as cross-validation—until a satisfactory set of items is 
obtained; 

. Standardizing, or fixing, the length of the test and the sequencing of 
items, as well as the administration and scoring procedures to be used 
in the final form of the test, on the basis of the foregoing analyses; 

. Administering the test to a new sample of individuals—carefully se- 
lected to represent the population of test takers for whom the test is 
intended—in order to develop normative data or performance criteria, 
indexes of test score reliability and validity, as well as item-level statis- 
tics for the final version of the test; 

. Publishing the test in its final 

form, along with an adminis- 


DON’T FORGET 


tration and scoring manual, ac- 
companying documentation of 
standardization data, reliability 
and validity studies, and the 
materials needed for test ad- 
ministration and scoring (see 
AERA, APA, & NCME, 1999, 
chap. 6). 


It is standard practice to categorize 
tests broadly into the areas of “ability” 
and “personality.” This traditional dis- 
tinction—repeatedly used in this 
chapter and elsewhere for the sake of 
convenience—rests on the notion 
that some tests are designed primarily 
to assess aspects of cognitive behavior 
whereas others are designed to assess 


For a test that is to be published 
commercially, these steps may take 
years and may have to be repeated 
several times if the initial results are 
less than adequate. In addition, most 
standardized tests are revised from 
time to time due to the gradual obso- 


aspects of behavior related to emo- 
tional functioning. However, when 
considering the topic of test items, 
and tests in their entirety, it is impor- 
tant to remember that cognitive and 
emotional factors are inseparable and 
that behavior samples reflect all as- 
pects of a person's functioning. 
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lescence of norms, performance criteria, and test content. Some standardized 
instruments used in large-scale testing, such as the SAT (formerly known as the 
Scholastic Aptitude Test), are almost continuously revised and refined. Obvi- 
ously, tests that are used on a limited basis, such as in classroom settings or in 
specific research studies, do not undergo such a rigorous process. Nevertheless, 
they still need to be designed with care according to preestablished specifica- 
tions, and their results must be psychometrically defensible—in terms of item 
characteristics, validity, and reliability—if they are to accomplish their purposes 
successfully. 


TEST ITEM TYPES 


The variety of items that make up psychological tests is immense. Such a variety 
defies easy categorization. Test items, like tests as a whole, can differ in terms of 
content and format, as well as in the medium through which they are adminis- 
tered, the manner in which they are scored, and the kind of processing that they 
call forth in test takers. One of the most basic distinctions among test items con- 
cerns the type of responses they require from test takers. From this standpoint, 
all test items can be classified into two broad categories, namely, selected- 
response items and constructed-response items. Tests designed to evaluate abili- 
ties as well as those intended for the assessment of personality use either one or 
both of these item types, depending on the nature of the behavior samples needed 
for the purpose of the test. Similarly, items of both types may be used either in 
group of in individual testing. Rapid Reference 6.2 provides information on 
where to obtain samples of various kinds of test items. 


Selected-Response Items 


Selected-response items, also known as objective ot fixed-response items, are close-ended in 
nature; they present a limited number of alternatives from which the test taker 
must choose. In ability tests, items of this type include multiple-choice, true-false, 
ranking, and matching, as well as items that call for a rearrangement of the op- 
tions provided. Typically, objective items in tests of ability are scored simply as 
pass-fail, although it is also possible to assign partial credit for certain response 
options. Examples of various kinds of selected-response ability items, both from 
standardized and teacher-made tests, will be recalled by anyone who has been 
schooled in the United States within the past few decades. Selected-response 
items were not always used as frequently in the United States as they are at pres- 
ent, nor are they used as frequently in all nations. In fact, much of the criticism 
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= fid Reference 6.2 


How To Locate Examples of Various Types of Test Items 


e The Educational Testing Service Web site (http://www.ets.org) provides links to 
several of its major testing programs. Sample items are available in the test 
preparation sections of the Web site for these programs. For instance: 


e The College Board Web site (http://www.collegeboard.com) provides SAT 
Verbal and Math mini-tests—with the kinds of items used in the real SAT— 
that prospective test users can take for free. 


e The Graduate Record Examinations (GRE) site (http://www.gre.org) offers a 
practice version of the GRE General Test. 


* Questions from the various sections of the Test of English as a Foreign Lan- 
guage (TOEFL) can be downloaded from the TOEFL site (http://www 
toefl.org). 


e Samples of items from a variety of ability and personality tests are available at 
http://www.schuhfried.co.at, which is the Web site for the Schuhfried Com- 
pany, an Austrian organization that markets software products for computer- 
ized test administration. 


e Pictures of the instruments used in many sensitivity and performance tests can 
be found in the Evaluation and Assessment catalog of the Lafayette Instrument 
Company, available at http://www.licmef.com/downloads.htm#cat 


e Test publishers’ printed catalogs provide descriptions of test content and often 
include sample ttems. Catalogs posted on the Internet tend to list and describe 
tests but usually do not include sample items. Prospective test users can obtain 
printed catalogs by contacting the test publishers (see Appendix B). 


aimed at standardized testing in education within the United States revolves 
around the pervasive use of selected-response test items—especially multiple- 
choice items—and their perceived weaknesses from the pedagogical point of 
view. Many critics of “standardized testing” use this term loosely, and incorrectly, 
as a synonym for tests that employ the multiple-choice item format (see, e.g., 
Mitchell, 1992; Sacks, 1999). 

In personality tests, objective items may be either dichotomous or polyto- 
mous. Dichotomous items require a choice between two alternatives (e.¢., true-false, 
yes-no, like-dislike, etc.), whereas po/ytomous items present the test taker with three 
ot more (usually an odd number such as 3, 5, or 7) alternative responses to a state- 
ment. These alternatives are typically scaled in terms of degree of acceptance (e.g., 
Like, indifferent, or dislike), intensity of agreement (e.g,, from strongly agree to strongly 
disagree), frequency (e.g,, from never to very offen), and so forth—with the midpoint 
usually signifying a neutral, uncertain, or middle-of-the-road position. 
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Forced-Choice 

Objective items that require test takers to choose which one of two or more al- 
ternatives is most or least characteristic of them are called forced-choice items. Each 
of the alternatives in a forced-choice set represents a different construct, but they 
are matched in terms of social desirability so that they appear equally attractive or 
equally unattractive to test takers. This kind of item is used mainly in multidi- 
mensional personality inventories (i.e., inventories designed to assess several per- 
sonality constructs) in order to control for the tendency of test takers to respond 
in the ditection they perceive as more socially desirable. However, forced-choice 
alternatives are often paired in such a way that each choice test takers make lim- 
its the possible range of their scores on another one of the constructs or traits as- 
sessed by the multidimensional test. When this is the case, the resulting scores are 
ipsative in nature and cannot be interpreted in a normative fashion. [psative scores 
are essentially ordinal numbers that reflect test takers’ rankings of the constructs 
assessed by the scales within a forced-choice format test. This means that the rel- 
ative magnitude of the scores on each of the scales in such a test can be gauged 
only in comparison to the other scores obtained by the same individual on the 
other scales of the test, rather than to scores obtained by the normative groups. 
Moreover, the forced-choice format cannot eliminate the influence of social de- 
sirability altogether and may even interfere with test-taking rapport (see Chapter 
7 for a definition of rappor). In spite of these problems, forced-choice items are 
still used, especially in interest inventories and in tests—such as the Myers-Briggs 
Type Indicator (MBT1)—whose primary aim is to classify individuals into mutu- 
ally exclusive categories. Some forced-choice format tests (e.g., the Jackson Vo- 
cational Interest Survey) avoid the problem of ipsativeness by pairing alternatives 
that are drawn from two different sets of parallel scales so that the range of scores 
in each scale is not constricted. 


Advantages of Selected-Response Items 
Objective items are by far the most popular and frequently used type of test item. 
Their advantages derive from the ease and objectivity with which they can be 
scored, which result in significant time savings and enhance test score reliability; 
the issue of scoring error is virtually inapplicable to items of this type, except 
through clerical mistakes. Moreover, selected-response items make efficient use 
of testing time because more of them can be administered within any given time 
period than is the case with constructed-response items. Although they can also 
be administered individually, most tests that use selected-response items are in- 
tended for group testing. 

All the responses to objective items can easily and reliably be transformed into 
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a numerical scale for scoring purposes, a fact that greatly simplifies the quantita- 
tive analysis of these items. In ability tests, correct and incorrect answers are usu- 
ally assigned values of 1 or 0, respectively; occasionally, variations, such as 2, 1, or 
0, are available for partial credit. In personality tests, dichotomous items are also 
scored 1 or 0, depending on whether the test taker’s response is or is not in the di- 
rection of the construct that the test is designed to assess. The alternatives pre- 
sented in polytomous or multiple-response items may be translated into various 
numerical scales, such as 5, 4, 3, 2, 1 or +2, +1, 0, —1, or —2, or reduced to a binary 
(1 or 0) scoring format by collapsing categories. 


Disadvantages of Selected-Response Items 

In spite of their advantages, selected-response items are more susceptible than 
constructed-response items to certain problems. In tests of ability the major 
problem connected to objective items revolves around the issue of guessing, The 
possibility of correct guessing is ever-present when responses simply have to be 
selected. In dichotomous items, such as true-false, the probability of guessing 
correctly is a substantial 50%. When test takers guess the correct answers to ob- 
jective items, the amount of error introduced into their scores varies depending 
on what factors (e.g., pure chance, partial knowledge, the wording of items, etc.) 
were responsible for the correct guesses. Similarly, incorrect answers to objective 
items can easily occur as a result of haste, inattention, carelessness, malingering, 
ot other chance factors unrelated to the test taker’s level of knowledge or ability 
in the area covered by the item. 

In personality testing, the intended goals of selected-response items can be 
easily subverted for an even greater number of reasons. These include not only 
random or careless responding, but also test-taking response sets that are either 
intentionally or unintentionally misleading. Depending on the context in which 
the testing takes place and the particular mental set of the test taker, personality 
test responses can mislead in either a positive or a negative direction. For ex- 
ample, individuals taking a personality inventory in the context of applying for a 
job would naturally choose to present themselves in a much more favorable light 
than would persons being tested to determine whether psychiatric illness may be 
used as a mitigating factor in determining culpability in a criminal trial. Clearly, re- 
sponses to objective personality test items can be more easily manipulated by test 
takers than responses to ability test items, which cannot be faked in a positive di- 
rection, except by cheating (for a thorough treatment of many aspects of cheat- 
ing on tests, see Cizek, 1999). Because of their vulnerability to distortion, many 
personality inventories use special sets of items, validity scales, or other devices 
specifically designed to detect misleading or careless responding. 
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All of the possibilities just outlined can diminish the reliability and validity of 
test scores. Whereas constructed-response items are also susceptible to some of 
these problems, guessing on constructed-response ability tests is more difficult 
and, therefore, less likely. Responding in misleading ways on projective tech- 
niques and other constructed-response personality assessment tools presents a 
greater challenge for test takers. Moreover, the relatively unstructured nature of 
those instruments is such that even when test takers consciously attempt to de- 
ceive they may be providing some useful information. 

Preparing selected-response items is a difficult and time-consuming task that 
requires specialized test development and item writing skills, in addition to great 
familiarity with the construct or subject matter with which the test deals. Poorly 
prepared objective items can inadvertently provide clues to test takers or be 
phrased in terms that work to the benefit or detriment of a subset of test takers. 
Carelessly written multiple-choice items, in particular, often include alternatives 
that are (a) grammatically incompatible with the stem of the item, (b) susceptible 
to various interpretations due to imprecise wording, or (c) so ludicrous that they 
can be easily dismissed. 

Finally, selected-response items are clearly less flexible than constructed- 
response items with regard to the possible range of responses. Therefore, they of- 
fer no opportunity for assessing characteristics that may be special or unique to 
an individual test taker or that lie outside the range of alternatives provided. 


Constructed-Response Items 


The essential characteristic of constructed-response items, also known as free-response 
items, is that they are open-ended. Their variety is limitless, because constructed 
responses may involve writing samples, free oral responses, performances of any 
kind, and products of all sorts. 

In ability tests, the most common type of constructed-response items are 
essay questions and fill-in-the-blanks. The only constraints pertinent to free- 
response items in psychological tests are the conditions imposed by the test in- 
structions. Thorough instructions and procedural rules are indispensable for the 
standardized administration and scoring of all tests, including free-response tests. 
Directions for administering constructed-response tests should include stipula- 
tions on matters such as (a) time limits; (b) medium, manner, or length of the re- 
quired response; and (c) whether access to materials or instruments pertinent to 
the test (e.g., textbooks, calculators, computers, etc.) is permitted. 

Interviews, biographical data questionnaires, and behavioral observations are 
tools for the assessment of personality that often rely on open-ended responses. 
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In personality testing proper, the use of constructed responses is limited mainly 
to projective techniques. These methods usually require test takers to respond to am- 
biguous stimuli in the form of pictures (including inkblots) or verbal materials, 
such as words or incomplete sentences. Some projective techniques call for self- 
expression through drawings or other kinds of performances. The basic idea in 
all of these methods—which originated and are used mainly in clinical settings — 
is to present test takers with tasks that have a minimal amount of structure so they 
may respond as freely as possible and, in the process, reveal significant aspects of 
their personalities. In contrast to inventories, surveys, and other such objective 
instruments designed to evaluate specific constructs or trait constellations related 
to personality, projective techniques provide a less focused, more global ap- 
proach to assessment. 

By and large, the advantages and disadvantages of constructed-response items 
are the opposites of those presented by selected-response items. They neverthe- 
less deserve mention. 


Advantages of Constructed-Response Items 

Even when they are not individually administered, constructed-response items 
provide richer samples of the behavior of examinees and allow for their unique 
characteristics to emerge. Open-ended items offer a wider range of possibilities 
and mote creative approaches to testing and assessment than selected-response 
items. Moreover, constructed-response tasks elicit authentic samples of test tak- 
ets’ behavior in specific domains, as opposed to mere choices among prepack- 
aged alternatives. If one wishes to evaluate writing skills, memory, mathematical 
knowledge, mechanical skills, leadership ability, or any other type of perfor- 
mance, actual samples of what an individual can do are the only unassailable stan- 
dard. 


Disadvantages of Constructed-Response Items 
The major disadvantages of constructed-response items are related to score reli- 
ability and, as a consequence, to validity as well (see the section on the relation- 
ship between reliability and validity in Chapter 4). These disadvantages stem from 
the way in which constructed responses are scored and from the practical limita- 
tions that responses of this type impose on the length of a test. 

Scoring constructed responses, both in ability and personality tests, is always 
a more time consuming and complex matter than scoring selected responses be- 
cause some degree of subjectivity is invariably necessary. Even when scoring rubrics 
(instructions that specify the criteria, principles, and rules to be used in scoring 
and that provide illustrative examples) are carefully prepared and applied, there is 
always the possibility that a response will be evaluated differently by different 
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scorers due to its uniqueness or to some other factor. Checking the reliability of 
scores assigned by different raters is an indispensable and costly aspect of using 
tests with constructed-response items. Although interscorer differences cannot 
be eliminated completely, they can certainly be minimized by means of thorough, 
explicit, and pilot-tested scoring procedures as well as by the proper training of 
scorers. 

Scoring constructed responses gathered from the projective tools used in per- 
sonality assessment poses a special challenge in that the subjectivity of the scorer 
can enter into play in more ways than it does in the scoring of constructed re- 
sponses in ability tests. In addition, projective techniques lend themselves more 
readily to the use of informal, and often idiosyncratic, methods of administration 
and scoring that can further weaken their psychometric integrity (see, e.g., Lan- 
yon & Goodstein, 1997, chap. 4). 

Test length is another factor that affects the reliability of scores from tests that 
use constructed responses. Because these responses require more time for admin- 
istration and scoring, the number of items that can be included in constructed- 
response tests is usually much smaller than it is in selected-response tests. As dis- 
cussed in Chapter 4, all other things being equal, shorter tests are more prone to 
content sampling errors and produce scores that are less consistent than those 
of longer tests. Thus, from the point of view of internal consistency as well, 
the scores of constructed-response tests tend to be less reliable than those of 
selected-response tests. 

An additional complication pertinent to constructed-response items concerns 
the matter of response length. Since longer responses contain more material than 
shorter ones, variations in the length of constructed responses can affect scores 
considerably. This is especially pertinent for projective techniques, because 
longer—or more elaborate—projective responses are likely to contain more 
scorable (i.e., psychologically significant) elements than shorter ones. Moreover, 
projective devices that allow for variability in the number, as well as the length, of 
responses pose yet another complicating factor in the investigation of their psy- 
chometric properties due to lack of uniformity across test takers. The Rorschach 
test is the preeminent example of this problem, as evidenced by the long-standing 
controversy regarding the impact of response productivity on Rorschach scoring 
and interpretation (see, e.g., Groth-Marnat, 1997, p. 399; Meyer, 1992). 


ITEM ANALYSIS 


In the past few decades, the field of test development and test design, as well as 
the techniques of item analysis, have been undergoing a transition that is taking 
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hold gradually but is fundamentally altering the nature of psychological tests. This 
transition is partly due to the ease and efficiency with which test data can be col- 
lected, stored, analyzed, retrieved, and disseminated with the use of computers. 
In addition, since the 1960s, the methodology provided by the new approaches 
to psychological test construction collectively known as item response theory 
(IRT) or latent trait theory has been steadily supplementing —and in some cases 
replacing—traditional methods of test construction and design. Though IRT 
methods can be, and are, used in developing paper-and-pencil as well as com- 
puter-based tests of fixed length, their most salient advantage over traditional 
methodology is that they allow a more flexible and efficient test format through 
computerized adaptive testing (CAT). In CAT, item sequences can be individually tai- 
lored to the test takers’ ability levels, or to the test takers’ positions on whatever 
trait the test is designed to assess, on the bases of prior responses. In the next sec- 
tions of this chapter, traditional item analysis procedutes are presented first, fol- 
lowed by a discussion of IRT methodology. 


Quantitative Methods of Item Analysis 


For psychological tests, in general, the most important aspect of quantitative 
item analysis centers on statistics that address item validity. The question that item 
validity indexes attempt to answer is whether a specific item carries its own 
weight within a test by eliciting information that advances the purpose of the 
test. Psychometricians usually refer to item validity statistics as indexes of item 
discrimination, because their role is to reveal the extent to which an item accurately 
differentiates among test takers with 

regard to the traits or behaviors that 


the test is designed to assess. For CAUTION 
tests of ability, in particular, item <o205c00n0sso cos cocacosonecesne onquadesoeseaceduoneconessocancenanéaonancaced 
analysis includes procedures de- The term discrimination has acquired a 


negative connotation in everyday us- 
age due to its frequent association 
acteristics of items that have a bear- with the unfair treatment of women 
ing on their validity, namely, sem and racial minority groups. 

difficulty and item fairness. All of these In contrast, within the field of psycho- 
metrics, discrimination is considered a 
See ne desirable feature for test items. It 
qualitatively as well as quantitatively. refers to the extent to which items 
Qualitative evaluation usually is car- elicit responses that accurately differ- 
entiate test takers along the dimen- 
sions that tests are designed to evalu- 
ate. 


signed to gauge two additional char- 


item characteristics can be evaluated 


tied out by subject matter experts 
who inspect the content of items 
with regard to their appropriateness 
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and difficulty level as well as to whether they reflect the objectives that were 
specified for the test. Item content is also examined from the point of view of its 
possible unfairness or offensiveness to any group of potential test takers. Quan- 
titative evaluation of item difficulty and discrimination is carried out through 
statistics that assess whether items perform the way they were intended to per- 
form when they are administered to the kinds of test takers for whom the test is 
designed. 


Item Difficulty 


The Role of Item Difficulty in Ability Testing 

Given the self-evident proposition that the difficulty level of a test as a whole 
is a function of the difficulty levels of the individual items that make up the test, 
it follows that an easy test is one that is made up of easy items and a difficult test 
is one made up of hard items. This apparently simple premise becomes a bit 
more complicated as soon as we consider that difficulty is a relative matter. 
How difficult a test item is depends not only on its intrinsic simplicity or ac- 
cessibility, but also on the ability level of the test taker. For example, the proper 
use of the verb éfre (to be)}—which is the most common verb in the French lan- 
guage—is a far easier test task for a student in an Advanced French class than 
for one who is beginning to study that language. Thus, in order to properly cal- 
ibrate the difficulty level of a test, indexes of the re/ative difficulty of items for 
one or mote relevant groups of test takers are needed. Test developers use these 
indexes to determine the appropriateness of the items for the population and 
purpose for which a test is designed, 
as well as to decide where items ate 
placed within a test. 


DON’T FORGET 


In much the same way that the frames 
of reference for test score interpreta- 
tion, discussed in Chapter 3, may be 


How Is Item Difficulty Gauged? 
During the initial stages of test devel- 


either normative or criterion- 
referenced, so can the difficulty of test 
items be determined on either an ab- 
solute or a relative basis. Both aspects 
need to be considered in the process 
of test construction and test develop- 
ment with specific reference to the in- 
tended population of test takers and 
purpose of the test. 


opment, when the pool of items is 
generated, test authors can gauge the 
difficulty of the items they create 
based on mote or less objective stan- 
dards delineated in the specifications 
that have been drawn up for the test 
ot based on criteria that are agreed 
upon by experts in the subject matter 
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ot cognitive skills covered in a test. For instance, one standard that may be applied 
to calibrate the difficulty of words is the frequency with which they are used 
within a language. Thus, in a vocabulary test, easy items are words that are em- 
ployed frequently by the users of the language in question whereas the most dif- 
ficult items consist of words that occur rarely and would be unfamiliar to most 
test takers. Similarly, in a test of arithmetic, individual items can be aligned in 
terms of difficulty based on the evident complexity of the operations they re- 
quire, such as multiplication of whole numbers versus multiplication of fractions, 
and so forth. 

Once a set of items is administered to one or more groups, quantitative in- 
dexes of item difficulty, which addresses this issue from a normative perspective, 
can also be obtained. In analyzing test items from the normative viewpoint, the 
essential piece of information used to determine item difficulty is the percentage 
of test takers who answer an item correctly, also known as proportion (or percentage) 
passing, ot p, for short. The higher the percentage passing, the easier the item is. 
Since the p values of items hinge entirely on the ability level of the groups to 
whom they are administered, the make-up of such groups is quite important and 
should reflect the make-up of the population for whom the test is intended. 

Percentage passing or p values are ordinal numbers that, like percentile ranks, 
do not represent equal units. For this reason, provided that the trait an item mea- 
sures can be assumed to be normally distributed, p values are often transformed 
into z values, using the Table of Areas of the Normal Curve (see Appendix C). 
Once p values are converted into z values, the relative difficulty levels of items can 
be compared across various groups by administering a common set of items— 
called anchor items—to two ot more groups. Formulas to estimate the difficulty 
levels of additional items across the groups in question can then be derived based 
on the established relationships among the anchor items. This procedure, known 
as absolute scaling, was developed by Thurstone (1925). It allows for the difficulty 
of items to be placed on a uniform numerical scale for samples of test takers at 
different ability levels, such as students in various school grades. Rapid Reference 
6.3 presents a simple numerical example of how this is accomplished using the re- 
sults of five items that were administered to two groups. Since all of the five items 
in the example have higher p values (and lower z values) for Group B than for 
Group A, we may surmise that Group B is functioning at a more advanced level 
than Group A in the ability or content area tapped by these items. Figure 6.1 por- 
trays the relative difficulty of the five items for the two groups and graphically 
demonstrates that the difficulty levels of the five items for the two groups corre- 
late strongly and positively. The kind of data presented in Rapid Reference 6.3 and 
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= fid Reference 6S 


Conversion of Item Difficulty From Proportion Passing (p) 
to Normal Curve Units (z) 


Item difficulty can be represented in normal curve units (z values), provided that 
the trait measured by an item is assumed to be normally distributed. 


The z value for an item is derived by locating the proportion who passed (i.e., its 
p value) in the Table of Areas of the Normal Curve (see Appendix C): p values 
above .50 are found in column 3 of the table and assigned the corresponding z 
values with a negative sign; p values below .50 are located in column 4 of the table 
and the corresponding z values are assigned a positive sign. If p = .50, the z value 
for the item is zero. 


Numerical example for five items administered to two groups: 


Item Group A Group A Group B Group B 
Number p Value? zValue? p Value? zValue? 
| .841 —1.00 894 =|,.25 
2 50 0.00 69 | —0.50 
3 067 +1.50 ASS +1.00 
4 023 +2.00 067 |50 
5 SUU —2.00 Ss =2,5)| 


» 


‘The p values represent the proportion of individuals in Group A and Group B who passed each 
item. 


bT 


[he z values for the easiest items are large and negative whereas those for the most difficult 
items are large and positive. 


Figure 6.1 can be used to estimate the difficulty levels of additional items for one 
group, based on their difficulty values for the other, by means of regression anal- 
ysis (see Chapter 5). These types of procedures are applied in equating tests and 
test scores via anchor tests and fixed reference groups (see Chapter 3). 


Item Difficulty Levels, Test Difficulty Levels, and Test Purpose 

For any given group of test takers, the average score on a test is the same as the 
average difficulty of its items. Thus, if the average percentage passing (p) for the 
items in a test is 80%, the average score on the test will be 80% as well. The sig- 
nificance of the relationship among item difficulty, test purpose, and the ability 
level of the population of test takers for whom the test is designed may be clari- 
fied by a few examples. 
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Group A: z Values 


Figure 6.1 Scatterplot of relative difficulty of five items for Groups A and B 
(see Rapid Reference 6.3) 


e Classroom achievement tests are designed to evaluate the extent to 
which the students in a class have mastered the content of a course. In most 
academic settings a grade within the 70 to 79% range is considered to be av- 
erage. In order to achieve this average, most of the items in classroom tests 
should be within the reach of the majority of students, provided they have 
mastered course content at a level that the instructor considers to be aver- 
age. Such tests may include some items—involving concepts that have 
been emphasized in the course—that the whole class answers correctly ( p 
= 1.00), even though such items do not differentiate among the test takers 
(see Table 6.1 later in this chapter). However, an item that no one answers 
correctly ( p = 0) is not desirable in a classroom test because it indicates that 
even the best students failed to grasp it. 

e On the other hand, a test designed to screen a large applicant pool in 
order to select the top 10% of individuals might have items that cluster 
around a p value of .10, or 10%. Such a test would be considered too diffi- 
cult for all except the most highly qualified applicants who possess an ex- 
tensive amount of the knowledge or skills that the test is designed to assess. 
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e Many tests of ability are designed to maximally differentiate individu- 
als within a given population in terms of a cognitive trait that is assumed to 
be normally distributed, such as general intelligence or verbal skills. In such 
tests the range of difficulty of items has to be sufficiently broad to accom- 
modate both the most and least capable individuals in the potential popu- 
lation of test takers, and the p value of items should cluster around .50 (or 
50%) to provide maximum differentiation among test takers. Items with 
extreme p values (i.e., close to 0 or 1.00) should be avoided in such tests be- 
cause they fail to differentiate among test takers and are therefore excess 
baggage. Furthermore, if individuals who belong to the population for 
whom a test of this nature is designed are able to pass either all of the items 
ot none of the items, their scores are indeterminate. As discussed in Chap- 
ter 3, when test items are too easy for a certain group, the test is said to have 
insufficient ceiling and its score distribution will be negatively skewed; when 
the items are too difficult for a group, the test has an inadequate floor and its 
score distribution is positively skewed. Figure 2.4 displays examples of 
skewed distributions. 


Distractors and Difficulty 

In tests that use multiple-choice items, the incorrect alternatives, or distractors, can 
have a great deal of influence on item difficulty in two major respects. In the first 
place, the number of distractors directly affects indexes of item difficulty because 
the probability of guessing correctly is higher when the number of choices is 
smaller. In addition, item difficulty is also affected by the caliber of distractors. An 
ideal multiple-choice item is one in which (a) the correct alternative is obvious to 
the test takers who know the answer and (b) the distractors appear equally plau- 
sible to those who do not know it. Such items are hard to construct. To the extent 
that the distractors seem obviously wrong, are poorly worded, or are much 
shorter or longer than the correct alternative, they provide clues that savvy ex- 
aminees who do not know the answer may use to narrow their choices and select 
the correct answer. In order to avoid these and other problems in devising multi- 
ple-choice items, test authors should follow the guidelines for item writing pro- 
vided in textbooks such as Haladyna’s (1999). After a test is administered, an anal- 
ysis of the distractors also ought to be conducted. Such analyses tally the number 
of test takers who choose each distractor. Careful examination of the frequencies 
with which various distractors are chosen by test takers of different ability levels 
setves to detect possible flaws in the items. Ifa test is still under development, dis- 
tractors that are not functioning adequately (e.g., those that are not chosen by 
anyone or those that are more often chosen by test takers of high ability levels) 
should be discarded and replaced. 
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Is Item Difficulty a Relevant Concept in Personality Testing? 

The tasks that make up personality tests may not be designed to evaluate cognitive 
functioning, but they do involve cognitive processes. In selected-response instru- 
ments, such as personality inventories and questionnaires, the relevant cognitive 
processes are related to the inventory taker’s ability to understand the items. 
Therefore, the vocabulary levels and reading skills of potential test takers need to 
be considered in devising those items. Projective tasks, on the other hand, involve 
a certain amount of proficiency in whatever mode the responses are to be ex- 
pressed. Most projective instruments require a modicum of skill in verbal expres- 
sion, drawing, or some other kind of performance. Thus, the relative difficulty or 
ease of projective tasks for various kinds of examinees must also be considered in 
the development, administration, and interpretation of these instruments. 


Item Discrimination 


Item discrimination refers to the extent to which items elicit responses that accu- 
rately differentiate test takers in terms of the behaviors, knowledge, or other char- 
acteristics that a test—or subtest—is designed to evaluate. For the vast majority 
of tests, discriminating power is the most basic quality that items must have in or- 
der to be included in a test. In the process of test development, item discrimina- 
tion indexes—also known as indexes of item validity—are obtained using some 
criterion or indicator of the test takers’ standing on the construct that the test as- 
sesses. Criteria employed for this purpose may be (a) internal criteria with respect 
to the test that is under development (i.e., total score on the test), (b) external cri- 
teria of the same kinds as those used to validate tests as a whole and described in 
Chapter 5 (e.g., age, education, membership in contrasted diagnostic or occupa- 
tional groups, etc.), or (c) combinations of both internal and external criteria. 


Item Validation Criteria 

The choice of criteria against which test items are validated depends on the pur- 
pose of the test. Ability tests require criteria related to the content areas or skills 
they assess; personality tests require criteria pertinent to traits or aspects of be- 
havior with which they deal. The quality and appropriateness of criteria used in 
validating test items have important consequences with respect to the selection 
of items that will be retained in a test and, consequently, on the reliability and va- 
lidity of the test scores. 

When criteria external to the test are used in validating items, the validity of 
scores on the test as a whole is enhanced; when the internal criterion of total test 
score is used to validate items, the homogeneity of the test increases and, hence, 
reliability indexes based on interitem consistency are enhanced. In the develop- 
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ment of tests that assess a single unidimensional trait such as vocabulary or de- 
pression, total score may be used to validate items. This practice is based on the 
assumption that all the items within such tests ought to correlate highly with the 
total score on the test and with each other. On the other hand, in the development 
of tests that assess complex and multifaceted constructs such as intelligence, 
items are validated against external criteria that are also more global. Since the 
items of those tests may be assessing different aspects of a complex construct, 
they do not necessarily have to correlate highly with one another and their degree 
of correlation with the total score may vary. Most intelligence scales, for instance, 
include a mixture of items tapping the various types of skills associated with that 
construct—such as verbal, numerical, spatial, and logical reasoning abilities— 
and provide composite scores that incorporate performance on all the item types 
and are validated against external criteria, such as educational achievement. In in- 
struments of this kind, items are usually grouped into subtests with homogeneous 
content which are scored separately (see Table 4.1 and Fig. 4.1 in Chapter 4). 

As we have just seen, even though both external validity and internal consis- 
tency are desirable goals in test construction, the nature of the constructs as- 
sessed by a test may not allow both goals to be realized concomitantly. In addi- 
tion to the limitations imposed by the purpose of the test, external validation of 
test items may also be impractical due to the unavailability or inaccessibility of ex- 
ternal criterion data. A typical example of this kind of situation is provided by 
items from teacher-prepared classroom tests, such as those presented in Table 
6.1. When conducting item analyses of these tests, teachers cannot use any crite- 
rion other than total test score because to do so would be unfair. Classroom tests 
are designed to evaluate mastery of the skills and content covered within a course 
and their scores are not supposed to be tied to any factor other than students’ 
mastery of the specified objectives. 


Item Discrimination Statistics 
All statistical procedures used to gauge the degree to which items discriminate in 
terms of a criterion require information on (a) item performance and (b) criterion 
standing for individuals in the samples from which the item discrimination statis- 
tics are extracted. The traditional statistics used for this purpose are of two types: 
the index of discrimination statistic (or D) and a variety of correlational indexes. 
The index of discrimination (D) is used primarily for items in tests of ability that 
are scored as pass or fail, but it can also be applied for analyzing the items of other 
tests that use binary scoring. In order to compute D, test takers must be classified 
into distinct criterion groups based either on their total scores on the test or on 
some external indicator of their standings on the construct assessed by the test. It 
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Table 6.1 Sample Item Analysis Data From a Classroom Test 


Percentage Passing (p value) 


Item Total Upper’ Lower’ D index Point Biserial 
Number Group Group Group (Upper — Lower) Correlation Ce) 


1 100% 100% 100% 0 0.00 
2 88% 100% 50% 50 0.67 
3 38% 100% 0% 100 0.63 
4 75% 50% 50% 0 0.13 
5 75% 50% 100% —50 —0.32 
6 13% 50% 0% 50 0.43 


“The upper and lower criterion groups are made up of students whose scores on the whole 
test were in the top and bottom 27%, respectively, of the score distribution. 

Point biserial is an index of the correlation between the performance of each one of the 
test takers on the dichotomously scored item (pass-fail) and their total scores on the test. 


is customary to create criterion groups by separating the test takers used for item 
validity analyses into two extreme groups, for example, those who are in the top 
and bottom thirds on the criterion measure. Once the upper and lower criterion 
groups are created, the percentage of individuals ( p) within each group who 
passed the item—or responded to it in whatever direction is keyed as indicative of 
the construct assessed by the test—is calculated. The żndex of discrimination is simply 
the difference in the percentage or proportion of test takers in the upper and lower 
criterion groups who pass a given item or answer it in the keyed direction; D can 
range from +100 to —100 (or from +1.00 to —1.00). For ability tests, positive dis- 
crimination indexes indicate that more individuals in the upper criterion group 
than in the lower criterion group passed the item and the most desirable values of 
Date those closest to +100. Negative D values indicate that the items in question 
discriminate in the opposite direction and need to be either fixed or discarded. 

Table 6.1 displays the item discrimination indexes for six items from a test ad- 
ministered to a psychological testing class. Item 1, the easiest one of the six, was 
passed by all the students (p = 100%) and Item 6, the most difficult one, was 
passed by only 13%. Item 3, passed by 38% of the students, was relatively difficult 
and was the most discriminating item among them, with a D value of 100. Items 4 
and 5 were relatively easy ( p = 75%) but of questionable value. Item 4 did not dis- 
criminate between the two extreme criterion groups at all and Item 5 actually had 
to be discarded because its D index of —50 indicated that the item discriminated in 
the wrong direction. 
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Correlation coefficients of various kinds 


DON’T FORGET can also express the relationship be- 


; À es ea tween performance on an item and 
Most indexes of item discrimination p 


are biased in favor of items of inter- criterion standing, and thus provide 
mediate difficulty. For instance, if the indexes of item discrimination. The 


percentage passing (p value) of an type of correlation coefficient chosen 


item for the total sample is extreme i 

(100% and 0%), there can be no dif- to calculate these indexes depends on 
ference in the p values of the upper the nature of the two variables that 
and lower criterion groups for that are to be correlated, which are the 


item and its D index is 0. On the other 
hand, when the p value for the total : 
group is 50%, it is possible for the D sures. For instance, when item scores 


ndex to reach its maximum value of ate dichotomous (e.g., pass-fail) and 
+100 if everyone in the upper crite- 


rion group and no one in the lower : , 
criterion group passes it. Thus, for all (e.g., total test score) the point biser- 
tests whose goal is to ascertain differ- ial (r) correlation coefficient is most 


ences among individuals in terms of often used. On the other hand when 
some ability, items that center around thei i ied : 

a 50% difficulty level are preferred. PATI S CONES andtne CLC OU mea. 

sute are both dichotomous, the phi 

(0) coefficient is used. Both the point 

biserial and phi coefficients of correlation can range from —1.00 to +1.00 and are 


item scores and the criterion mea- 


the criterion measure is continuous 


interpreted in the same way as the Pearson z. Formulas for computing point bi- 
serial and phi coefficients, and several other types of correlation coefficients used 
in the analysis of item discrimination, are available in most basic statistics text- 
books. In any case, high positive correlations indicate a direct and strong rela- 
tionship between item and criterion, high negative correlations indicate an in- 
verse and strong relationship between item and criterion, and low correlations 
indicate a weak relationship between the two. Table 6.1 also lists the point biser- 
ial correlations between items and test scores for each of the six items discussed 
earlier. 


A Note About Speed 

Whenever tests of ability have time limits, speed of performance affects test 
scores to some extent. This topic was discussed in Chapter 4, in connection with 
the problems speeded tests pose in the computation of split-half reliability coef- 
ficients, and it needs to be considered in relation to item statistics as well. With re- 
gard to speed, tests can be classified into three types: pure speed tests, pure power 
tests, and tests that blend speed and power. 


e Pure speed tests simply measure the speed with which test takers can per- 
form a task. In a pure speed test, difficulty is manipulated mainly 
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through timing. These tests have items whose difficulty levels are uni- 
form and well within the capabilities of individuals who are likely to 
take the test, but time limits ate so short that most test takers cannot 
complete all the items. Thus, in most cases, the total score on a pute 
speed test is simply the number of items completed by the test taker. If 
test takers finish all of the items in a pure speed test, their actual capac- 
ity has not been determined because there is no way of knowing how 
many mote items they might have completed if more items were avail- 
able. 

© Pure power tests, on the other hand, have no time limits. In these tests, 
difficulty is manipulated by increasing or decreasing the complexity of 
items. Their difficulty range needs to be sufficiently wide to accommo- 
date the ability levels of all potential test takers. In power tests, items are 
arranged in ascending order of difficulty so that all test takers are able 
to complete at least some items, but the most difficult items are usually 
beyond the reach of the majority of test takers. A perfect score in a pure 
power test suggests that the test taker’s ability level exceeds the diffi- 
culty level of the most difficult items. In such cases, the test taket’s ac- 
tual level of ability is indeterminate due to the test’s insufficient ceiling. 

e Most ability tests fall somewhere between the extremes of the pure- 
speed/pure-power continuum. Their time limits typically allow test tak- 
ers to reach and attempt all or most of the items. As discussed previ- 
ously, the specific range and average of difficulty levels of ability test 
items depend on the purposes for which the tests are employed. 


In any test that is closely timed, the p values and discrimination indexes of 
items are a function of their position within the test rather than of their intrinsic 
difficulty or discriminant validity. This is so because items in the latter part of a 
test in which speed plays a significant role are attempted by fewer test takers, and 
those who attempt such items tend to be either the most capable test takers or 
those who rush through a test by responding randomly. As a result, the difficulty 
and discrimination indexes for items that occur late in speeded tests are likely to 
be misleading and special strategies need to be implemented in order to gain in- 
sight into the specific roles of speed and difficulty in such tests. 


Combining Item Difficulty and Item Discrimination 


In light of the interrelationship between item difficulty and item discrimination, 
the development of most tests of ability requires analyses that combine both of 
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these item characteristics. There are two approaches to achieving this end. The 
older methods consist of analyzing item-test regression and the more recent ones 
involve the use of item response theory (IRT). 


Item-Test Regression 

To perform item-test regression analyses it is necessary to calculate the propor- 
tion of individuals at each total score level who passed a given item. Table 6.2 pre- 
sents sample data of this kind for two items of a hypothetical ten-item ability test 
on which the total scores range from 1 to 10. Figure 6.2 displays the item-test re- 
gressions for both items, plotted from the data in Table 6.2. Item-test regression 
graphs combine information on item difficulty and item discrimination and allow 
us to visualize how each item functions within the group that was tested. If we as- 
sumed that the item statistics presented in Table 6.2 were based on a large and 
representative sample of test takers, these data would make it possible to evaluate 
the two items and draw certain conclusions about them, as described in the fol- 
lowing paragraphs. 


© [tem 1 is easier than Item 2, because its 50% threshold is lower. The 50% 
thresholds are represented in Figure 6.2 by perpendicular broken lines 
that have been drawn from the points where the regression graphs for 
each item meet the horizontal line at p = .50 down to the baseline that 
displays total scores on the test. For Item 1, the 50% threshold is at the 
point where total score equals 4, whereas for Item 2, it is where total 


Table 6.2 Item-Test Regression Data for Two Items 


Proportion of Examinees Who 
Answered Each Item Correctly 


Total Score Item 1 Item 2 
10 1.00 1.00 
9 .60 85 
8 75 .70 
7 65 .50 
6 .70 .45 
5 .80 .30 
4 .50 .00 
3 .40 .00 
2 .30 .00 
1 .35 .00 
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Proportion of Correct Answers 
(ej 
O 


1 2 3 4 5 6 7 8 9 10 
Total Score on the Test 


ITEM 1 O———O ITEM 2 & — —4 


Figure 6.2 Item-test regression for Items | and 2 (see Table 6.2) 


score equals 7. These data suggest that the level of ability needed to 
have a 50-50 chance of passing Item 1 is lower than the level of ability 
needed to have an equal chance of succeeding on Item 2. 

© Item 2 discriminates better than Item 1. The item-test regression is steeper for 
Item 2 than for Item 1 and shows no reversals of direction in the pro- 
portion passing the item at each total score point. In contrast, Item 1 
shows a more gradual item-test regression and four reversals in its direc- 
tion (at the total score points of 2, 6, 7, and 9). Since total score on the 
test is assumed to reflect a test taker’s level of ability, the item-test regres- 
sions in Figure 6.2 suggest that the relationship between ability and item 
performance is more direct and stable for Item 2 than it is for Item 1. 

© [tem 1 is more likely than Item 2 to be answered correctly by guessing. This infer- 
ence is based on the fact that the proportion of correct answers to Item 
1 is fairly high (.35) even for those individuals who obtained a total 
score of 1, which was the lowest score on the test. In contrast, no one 
with a test score below 5 was able to answer (or guess) Item 2 correctly. 


238 ESSENTIALS OF PSYCHOLOGICAL TESTING ) 


* Conclusion. Altogether, examination of the item-test regression data pre- 
sented in Table 6.2 and Figure 6.2 suggests that (a) Item 2 is more diffi- 
cult than Item 1, (b) Item 2 appears to function better than Item 1 in 
terms of its power to discriminate among individuals with low and high 
scores on the hypothetical set of ten ability items, and (c) Item 2 is more 
impervious to guessing than Item 1. 


Although these analyses of item-test regression are informative, they are fairly 
crude and quite dependent on the samples and item sets from which the data are 
obtained. Item response theory uses the same kinds of empirical data involved in 
item-test regression analyses as a point of departure for far more sophisticated 
forms of item analyses and more ambitious test development strategies. 


ITEM RESPONSE THEORY 


The label stem response theory (IRT) refers to a wide, and growing, variety of mod- 
els that may be used to design or develop new tests and to evaluate existing tests. 
IRT models differ in the mathematical formulas they employ, in the number of 
item characteristics that they may account for, and in the number of trait or abil- 
ity dimensions they specify as the objectives of measurement. In addition, differ- 
ent methods are used depending on whether item data are dichotomous (pass- 
fail, true-false, etc.) or polytomous (i.e. consisting of multiple response 
categories). The procedures encompassed within IRT are extensive and complex. 
Until fairly recently, published presentations of these methods were too difficult 
to understand without a solid grasp of mathematics and statistics. Fortunately, in 
the past few years a number of excellent and more accessible materials on IRT 
techniques have been published. Rapid Reference 6.4 lists a selection of some of 
the most useful resources available. 


Classical Test Theory Versus Item Response Theory 


The label classical test theory (CTT) is used, in contradistinction to IRT, to refer to 
all of the traditional psychometric methods of developing and evaluating tests 
that predate IRT. The fundamental methods of CTT were developed in the early 
part of the 20th century and were well established by the middle of that century. 
They were perhaps best summarized in Gulliksen’s (1950) classic volume on the 
theory of mental tests but have been described before and since then in countless 
other sources. The psychometric principles and procedures of CTT have been 
continuously refined and expanded; they are still widely used and will continue to 
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= fid Reference 6.4 


Sources of Information on Item Response Theory and 
Computer Adaptive Testing 


Books 


Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental mea- 
surement in the human sciences. Mahwah, NJ: Erlbaum. 

Embretson, S. E., & Reise, S. P (2000). Item response theory for psychologists. 
Mahwah, NJ: Erlbaum. 

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item 
response theory. Newbury Park, CA: Sage. 


Wainer, H. (2000). Computer adaptive testing: A primer (2nd ed.). Mahwah, NJ: 
Erlbaum. 


Internet Resources 


Many of the item response theory (IRT) resources previously available at the 
now-defunct online Educational Resource Information Center (ERIC) Clearing- 
house on Assessment and Evaluation can be found at http://edres.org/irt, a Web 
page maintained by Lawrence Rudner the former director of the ERIC Clearing- 
house on Assessment and Evaluation. Through this Web site one can access many 
useful IRT-related materials, such as the following: 


An excellent tutorial on item response theory, made available by the University 
of Illinois at Urbana-Champaign's IRT Modeling Lab; 


The second edition of Frank Baker's classic book The Basics of Item Response 
Theory (2001); 


Links to free and commercially available IRT software, as well as to paper col- 
lections and books on IRT. 


CAT Central is a Web site that has a variety of resources for research and appli- 
cations of computerized adaptive testing (CAT), including basic information on 
CAT, an extensive bibliography, a listing of major testing programs that employ 
CAT, and links to other CAT-related resources. CAT Central can be found at the 
following address: 


http://www.psych.umn.edu/psylabs/CAT Central 


be used for the foreseeable future. In fact, most books on psychological testing — 


including the present one—deal largely with CTT. Some of the major contrasts 


between CTT and IRT were mentioned briefly in Chapter 3 in connection with 


the topic of test equating. However, the range and significance of the changes en- 


tailed in the transition between the conventional procedures of CTT and the 


model-based approach to measurement that characterizes IRT extend well be- 


yond that topic. 
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At present, IRT methods are em- 


DON’T FORGET ployed in a more limited range of in- 


, i struments than the traditional meth- 
The sections on item response theory 


(IRT) and computer adaptive testing ods of CTT. This is due partly to the 
(CAT) in Chapter 3 provide a basic significant assumptions IRT re- 
introduction to some of the distinctive 
features of these relatively novel ap- 
proaches to psychological measure- 
ment. Readers may find it useful to and partly to the more extensive data 
look back at those earlier sections as a collection efforts needed in ordet to 
prelude to the material on those top- 
ics presented in this chapter. 


quites—concerning item responses, 
latent traits, and their relationships— 


calibrate items using IRT models. 
Moreover, in contrast to the well- 
established, comparatively simple, 
and widely used techniques of CTT, IRT methods are still evolving, considerably 
more sophisticated from the mathematical standpoint, and unfamiliar even to 
many testing professionals. As Embretson (1996, 1999) has made abundantly 
clear, even though CTT and IRT share some conceptual foundations and there is 
a certain amount of reciprocity between the two approaches, many of the tradi- 
tional rules of measurement implicit in CTT must be revised or abandoned when 
IRT models ate applied to measurement tasks. Rapid Reference 6.5 presents one 
of the several contrasting features of the two approaches. 

One of the most basic differences between CTT and IRT stems from the fact 
that in CTT, interest centers mainly on the examinee’s total score ona test, which 
represents the sum of the item scores; whereas in IRT—as its name implies—the 
principal focus is on the examinee’s performance on individual items. In IRT, the 
cateful development and calibration of test items in terms of the information they 
provide about a specific psychological construct is a primary concern. To ac- 
complish this calibration IRT relies on mathematical models of the relationships 
between abilities —or whatever other unobservable constructs (i.e., latent traits) 
a test is designed to assess—and responses to individual items. 

Broadly speaking, the goals of IRT are (a) to generate items that provide the 
maximum amount of information possible concerning the ability or trait levels of 
examinees who respond to them in one fashion or another, (b) to give examinees 
items that are tailored to their ability or trait levels, and thus (c) to reduce the num- 
ber of items needed to pinpoint any given test taker’s standing on the ability or la- 
tent trait while minimizing measurement error. Reducing the number of items in 
a test by selecting those that are most appropriate to the test taket’s level of abil- 
ity—without a consequent loss of reliability—is an important goal in group test- 
ing. This is especially true for testing programs that are carried out on a massive 
scale, such as the SAT. A reduction in the number of items administered saves 
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Classical Test Theory Versus Item Response Theory: 
A Contrast on the Matter of Test Length and Reliability 


The new rules of measurement, described by Embretson (1996, 1999), highlight 
some crucial differences between classical test theory (CTT) and item response 
theory (IRT). Among these is the contrast between the old rule that “longer tests 
are more reliable than shorter tests” and the new rule that “shorter tests can be 
more reliable than longer tests” (p. 343). To wit: 


e As discussed in connection with split-half reliability and the Spearman-Brown 
formula (Chapter 4), CTT holds that, all other things being equal, a larger num- 
ber of observations will produce more reliable results than a smaller number of 
observations. If the length of a test increases, through the addition of parallel 
items, the proportion of true variance to error variance also increases and, 
hence, so does test score reliability. Thus, for two comparable tests of fixed 
engths (e.g., 50 vs. 40 items), the scores on the longer test will be more reliable 
than those on the shorter test. 


e In the computer adaptive testing (CAT) that IRT methods allow, item selection 
is optimally suited to test takers’ levels on the trait being assessed. Inappropri- 
ate items (e.g,, those that are too easy or too difficult for the test taker) are 
eliminated, resulting in a shorter test. Because IRT methods also calibrate the 
information that is obtained from each item response more precisely, measure- 
ment error can be reduced and reliable scores can be obtained from fewer, but 
more informative, item responses. 


e For more detailed explanations of these notions, along with numerical and 
graphic illustrations, see Embretson (1996, 1999) and Embretson and Reise 
(2000). 


time, saves money, and minimizes the frustration test takers experience when 
confronted with items that are not suited to their ability levels. Indeed, in large- 
scale testing programs, CATs developed through IRT methods are gradually re- 
placing tests of fixed length in paper-and-pencil and computer-administered for- 
mats (Embretson & Reise, 2000). 


Shortcomings of Classical Test Theory 

Classical test theory and IRT differ in many other respects. Although a full dis- 
cussion of these differences is beyond the scope of this volume, a few need to be 
mentioned because of their significance with regard to test development and the 
analysis of items. One way to contrast the two methodologies is by outlining the 
shortcomings of CTT that IRT attempts to overcome. Although some of these 
points have been mentioned earlier in this volume, they are reiterated here more 
fully and in specific reference to the comparison between CTT and IRT. 
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e CTT indexes of item difficulty and item discrimination are group- 
dependent: Their values may change when computed for samples of test tak- 
ers who differ from the ones used for the initial item analyses in some as- 
pect of the construct being measured. In contrast, the estimates of item 
characteristics obtained through IRT methods are assumed to be invariant 
and provide a uniform scale of measurement that can be used with differ- 
ent groups. 

* For tests of fixed length developed using CTT methods, the trait or 
ability estimates (i.e., the scores) of test takers are /est—dependent. In other 
words, scores are a function of the specific items selected for inclusion in a 
test. Therefore, comparisons of scores derived from different tests or dif- 
ferent item sets are not possible unless test equating procedures, which are 
often not feasible, are used (see Chapter 3). Moreover, even when equating 
procedures are applied, the comparisons that can be made are limited to the 
tests that were equated. In the case of IRT— provided that the data fit the 
model and provided that certain assumptions are met—estimates of abili- 
ties or traits are independent of the particular item set administered to ex- 
aminees. Instead, trait estimates are linked to the probabilities of exami- 
nees’ item response patterns. 

e In CTT methodology, the reliability of scores (i.e., trait or ability esti- 
mates) is gauged by means of the standard error of measurement (SEM), 
which is assumed to be of equal magnitude for all examinees (see chap. 4). 
In fact, since score reliability depends on how well suited test items are to 
examinees’ trait or ability levels, and since trait levels are not equal across 
examinees, this assumption is not plausible for traditional tests. On the 
other hand, when IRT methodology is combined with adaptive testing pro- 
cedures, the standard errors of trait or ability estimates resulting from a test 
administration depend on the particular set of items selected for each ex- 
aminee (see Rapid Reference 6.5). As a consequence, these estimates vary 
appropriately at different levels of the trait dimensions and convey more 
precise information about the reliability of measurement. 


Essential Features of Item Response Theory 


Since most IRT models currently in use are unidimensional models, the present 
discussion is limited to those. Unidimensional IRT models assume (a) that the items 
comprising a test or test segment measure a single trait and (b) that the item re- 
sponses of test takers depend only on their standing with regard to the trait being 
measured. As Rapid Reference 6.6 suggests, from a realistic viewpoint, neither 
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What Makes the Behavior Samples Gathered Through 
Test Items So Complex? 


e Regardless of what construct test items are meant to assess, they always involve 
multiple dimensions. To begin with, some ability to attend to test stimuli is re- 
quired in order to respond to any test task, as is a modicum of short-term 

memory. In addition, all test items involve a specific content, format, and 

medium, and require a specific set of cognitive skills. For instance, depending on 
its mode of presentation and response, a simple vocabulary item may involve 
reading, writing, spelling, oral comprehension, verbal expression, logical reason- 
ing ability or knowledge of etymology, not to mention attention, memory, and 
possibly speed as well. 


e Test takers are complex and unique beings. They bring a combination of fac- 
tors—such as genetic endowment, experiential histories, developed skills, 
traits, habits, and attitudes, as well as transitory physiological and emotional 
states—to bear on test tasks. Because responses to test items are a function of 
test takers’ unique blend of all the elements they bring to the tasks, such re- 
sponses are never equivalent in every respect. For example, test items that re- 
quire a series of arithmetical computations present a bigger problem for a test 
taker who experiences math anxiety than for one who does not, even if both 
are equally capable of performing the computations in a non-test situation. 


one of these assumptions can ever be fully met. However, when all the items in a 
test or test segment are designed to measure a single predominant trait, the as- 
sumptions of unidimensional models can be met adequately enough to make the 
models workable. 

In the next few paragraphs, some of the features common to most of the IRT 
models currently in use are summarized to give readers a general idea of how IRT 
methodology is applied to the calibration of test item data. For the sake of brevity 
and simplicity, this presentation avoids the use of mathematical formulas and 
concepts that are not essential to a basic understanding of the fundamental ideas 
of IRT. Interested readers can find more extensive treatments of these methods, 
as well as explanations of their mathematical bases, in the soutces listed in Rapid 
Reference 6.4. 


* In IRT, models are based on the premise that a person’s performance 
on any test item is a function of, and can be predicted by, one or more traits 
ot abilities. The models seek to specify the expected relationships between 
examinees’ (observable) responses to items and the (unobservable) traits 
that govern their responses. Because they entail predictions, IRT models 
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can be evaluated (i.e., confirmed or rejected) depending on how well they 
fit the data derived from responses to test items. 

° IRT methods employ test and item response data from large samples 
known to differ on the ability or personality trait that the test under devel- 
opment is designed to assess. Such samples need not be representative of a 
defined population, but they must include groups of individuals who are at 
different levels in the trait or ability continuum. In addition, the items in the 
initial pool need to be carefully constructed or selected for their potential 
as indicators of the trait to be assessed. 

e After item and test score data are collected, they are used to derive es- 
timates of item parameters that will place examinees and items along a com- 
mon scale for the ability or trait dimension. [tem parameters are the numeri- 
cal values that specify the form of the relationships between the abilities or 
traits being measured and the probability of a certain item response. For in- 
stance, item difficulty parameters express the difficulty of an item in terms of 
the ability scale position where the probability of passing the item is .50. 
Table 6.3 displays a small, hypothetical set of raw data on ten dichoto- 
mously scored items administered to ten individuals (A through J). Al- 
though a realistic example would include a much larger and mote varied 
sample of test takers—possibly grouped into categories based on their to- 
tal scores, instead of individually—the data in Table 6.3 illustrate the kind 
of information that may be used to estimate item difficulty parameters in 
relation to ability levels. Item parameters are obtained through a variety of 
procedures that require the use of specialized computer programs (see, e.g., 
Embretson & Reise, 2000, chap. 13). These procedures employ nonlinear 
mathematical functions, such as logistic functions, which yield item char- 
acteristic curves (see below). Nonlinear mathematical models are necessary 
because the linear regression model, discussed in Chapters 2 and 5, is not 
suitable for describing how changes in trait levels relate to changes in the 
probability of responding to an item in a specific way. 

° An item characteristic curve ICC) is the graphic representation of a 
mathematical function that relates item response probabilities to trait lev- 
els, given the item parameters that have been specified. For instance, the 
ICC of a dichotomous ability test item visually expresses the expected rela- 
tionship between ability level and probability of passing an item. In the case 
of personality test items, ICCs display the expected relationship between 
trait levels and probability of responding to an item in a specific manner. 
The hypothetical ICCs presented in Figure 6.3 exemplify the three most 
common unidimensional logistic models for dichotomous item response 
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Table 6.3 Hypothetical Example of Raw Item and Person Data Used in 
IRT Parameter Estimation 


Item 

Person 1 2 3 4 5 6 7 8 9 10 Total 
A 1 1 1 0 0 1 1 1 1 8 
B 0 0 1 1 1 0 0 0 0 4 
C 0 0 1 1 1 0 0 0 0 4 
D 1 1 0 0 1 0 0 0 1 1 5 
E 1 1 1 1 1 1 1 1 1 10 
F 1 1 1 0 0 0 0 1 1 6 
G 1 1 0 0 1 0 1 1 1 7 
H 0 1 1 0 0 0 0 0 0 3 
I 0 0 1 0 0 0 0 1 1 1 4 
J 0O 1 1 0 0 0 0 1 1 5 

Total 5 H 8 8 4 4 2 4 7 7 


Note: The 1’s and O’s in each cell indicate whether Persons A to J passed or failed each of 
the 10 items. Test takers’ total scores, in the last column, can be used to calculate ability es- 
timates; item total scores, in the last row, can be used to calculate item difficulties. 


data. Panel A of Figure 6.3 displays ICCs for the one-parameter logistic 
model, also known as the Rasch model in honor of the mathematician who 
developed it (Rasch, 1960/1980). Panels B and C of Figure 6.3 portray the 
two- and three-parameter logistic models, respectively. 


e Panel A of Figure 6.3 displays the ICCs for two items that differ only with 
respect to difficulty. Item 1 is easier than Item 2. The location of the 
difficulty parameter (i.e., the ability level associated with a .50, or 50%, 
probability of success) is lower for Item 1 (X) than for Item 2 (X;). 
Since the slopes of the two curves are the same, we can infer that the 
two items function equally well in terms of the relationship between 
ability and probability of success throughout the ability scale. 

© Panel B of Figure 6.3 shows ICCs for two items that differ in two parame- 
ters, namely, difficulty and discrimination. In this instance, the ability 
level associated with a 50% probability of success is somewhat higher 
for Item 1 (X) than for Item 2 (X,). Furthermore, the slopes of the two 
curves—which show the ratio of change in ability to change in proba- 
bility of success for each item—are different and cross over at a certain 


1.0 
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of 0.5 
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B. 
1.0 
Probability 
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Figure 6.3 Item characteristic curves: A, one-parameter; B, two-parameter; 
and C, three-parameter models 
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point. This configuration suggests that the two items function differ- 
ently in terms of their relation to the ability trait and do not discrimi- 
nate equally well at all points in the ability scale. In a truly unidimen- 
sional test, items with ICCs that intersect—such as those in Panel 
B—are undesirable. 

© Panel C of Figure 6.3 displays the ICCs for two items that differ along 
three parameters, namely, difficulty, discrimination, and probability of 
chance success (or guessing). The ICC for Item 1 has a steeper slope 
than the curve for Item 2, and shows a steady rise in the probability of 
success as ability levels increase up to a certain point. In contrast, Item 
2 clearly does not discriminate among individuals at different ability 
levels as well as Item 1: Its ICC shows a less pronounced relationship 
between ability level and probability of success. Note also that the ICC 
for Item 2 shows a fairly high probability of success even at the lowest 
end of the ability spectrum. This suggests that test takers at low levels 
of ability are able to correctly guess the answer to Item 2. Moreover, the 
50% success probability is associated with a higher level of ability (X,) 
for Item 2. Clearly, an item with an ICC like that of Item 2 in Panel C 
would be less efficient than Item 1 from a measurement perspective. 

e As is the case with any theoretical model, the extent to which the as- 
sumptions of IRT models are met can be gauged by comparing their pre- 
dictions against empirical data and evaluating the magnitude and signifi- 
cance of any discrepancies that are found between the observed data and 
the predictions of the models. If the fit between the ICC model-based ex- 
pectations and the performance of examinees on a test item is sufficiently 
close, IRT parameters are used to derive the item’s information function. 

° An item information function re- 
flects the contribution an item 


makes to trait or ability estimation DON’T FORGET 


at different points in the trait Ot pine ieee ee 
In item response theory (IRT), item 
parameters are assumed to be invari- 

ant for the population, meaning that 
whether and where to incorporate they should be stable even when they 
items into a test, in light of the are computed on groups differing in 

the ability or trait being measured. 

Thus, unlike the item analysis statistics 


ability continuum. Item informa- 
tion functions help to decide 


test’s objectives. Trait-level esti- 


mates that locate test takers on the described earlier in the chapter IRT 
trait dimension are derived from parameters are not linked to the per- 
formance of any given reference 


their specific patterns of successes heii 
or failures on a series of items. In 
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IRT, the zest information function, which is the sum of item information func- 
tions, corresponds to the CTT notion of score reliability (see Chapter 4). 
Test information functions are calculated and used to obtain standard er- 
rors of estimation at each level in the trait or ability scale. These standard 
errors, in turn, create confidence bands for the ability estimates in a fash- 
ion similar to the way standard errors of measutement in CTT are used to 
create confidence bands for obtained scores. 


As may be gathered from the discussion of the ICCs presented in Figure 6.3, 
even the unidimensional IRT models exemplified in that figure can become quite 
complex, as the number of parameters encompassed in the models increases. Ap- 
plying IRT models to the development of instruments aimed at assessing broader 
and more contentious intellectual and personality constructs is a much more dif- 
ficult proposition (see, e.g., Reise & Henson, 2003). Multidimensional IRT mod- 
els—which assume that two or more traits contribute to item responses—are 
now being used to explore and explain more complex and multifaceted con- 
structs. Some of these newer, and more complicated, models are described by 
Embretson and Reise (2000). 


Item Fairness 


Generally speaking, there are many ways in which test items, as well as tests, can 
be biased or unfair to individual test takers or groups of test takers. As far as tests 
are concerned, the possibility of bias can be investigated by ascertaining whether 
test scores have the same meaning for members of different subgroups of the 
population (see Chapter 5). The question of test fairness, on the other hand, is a 
more complex and controversial issue. Whereas there is general agreement that 
unfair uses of tests must be avoided, exactly what constitutes fairness in testing is 
a matter of considerable debate (AERA, APA, NCME, 1999, pp. 74—76). Never- 
theless, test users have a major responsibility in implementing fair testing prac- 
tices through a thoughtful consideration of the appropriateness of instruments 
for their intended purposes and for potential test takers (Chapter 7). 

At the level of test items, questions concerning bias and unfairness are more 
circumscribed and are usually taken up while a test is under development. To this 
end, test items are analyzed qualitatively and quantitatively throughout the pro- 
cess of test construction. Naturally, the extent to which test items undergo these 
reviews is related to the intended purpose of a test. Special care is taken to elimi- 
nate any possible bias or unfairness in the items of ability tests that are to be used 
in making decisions that have significant consequences for test takers. 


< ESSENTIAL TEST ITEM CONSIDERATIONS 249 


Qualitative Analysts of Item Bias 

The qualitative evaluation of test items from the point of view of fairness is based 
on judgmental procedures conducted by panels of demographically heterogeneous 
individuals who are qualified by virtue of their sensitivity to such matters and, 
preferably, by their expertise in the areas covered by a test as well. Typically these 
reviews occur at two stages. During the initial phase of test construction, when 
items are written or generated, they are examined in order to (a) screen out any 
stereotypical depictions of any identifiable subgroup of the population, (b) elimi- 
nate items whose content may be offensive to members of minority groups, and (c) 
ensure that diverse subgroups are appropriately represented in the materials con- 
tained in an item pool. In this initial review, individuals who are familiar with the 
linguistic and cultural habits of the specific subgroups likely to be encountered 
among potential test takers should also identify item content that may work to the 
benefit or detriment of any specific group, so that it may be revised. The second 
stage of qualitative item review occurs later in the process of test construction, af- 
ter the items have been administered and item performance data have been ana- 
lyzed separately for different subgroups. At this stage, items that show subgroup 
differences in indexes of difficulty, discrimination, or both are examined to identify 
the reasons for such differences and are either revised or discarded as warranted. 


Quantitative Analysts of Item Bias 
The quantitative assessment of item bias has sometimes been linked simply to the 
differences in the relative difficulty of test items for individuals from diverse de- 
mographic groups. However, this interpretation of item bias is viewed as naive by 
testing professionals who do not consider differences in the relative difficulty of 
an item for different groups to be sufficient evidence that the item is biased (see, 
e.g., Drasgow, 1987). Instead, from a psychometric standpoint, an item is con- 
sidered to be biased only if individuals from different groups who have the same 
standing on a trait differ in the probability of responding to the item in a speci- 
fied manner. In tests of ability, for instance, bias may be inferred when persons 
who possess equal levels of ability, but belong to different demographic groups, 
have different probabilities of success on an item. Thus, in the testing literature, 
item bias is more properly described as differential item functioning (DIF ), a label 
that more pointedly denotes instances in which the relationship between item per- 
formance and the construct assessed by a test differs across two or more groups. 
Classical procedures for the quantitative analysis of DIF involve comparisons 
of the item difficulty and item discrimination statistics for different groups. For 
example, if a test item has a low correlation with total test score (i.e., poor dis- 
crimination) and is more difficult for females than for males, it would obviously 


250 ESSENTIALS OF PSYCHOLOGICAL TESTING ) 


be suspect and should be discarded. However, the analysis of DIF by means of 
simple comparisons of the item-test correlations and p values for different 
groups is complicated by the fact that groups of various kinds (e.g., sex groups, 
ethnic groups, socioeconomic groups, etc.) often differ in terms of their average 
performance and variability, especially on ability tests. When group differences of 
this kind are found in the distributions of test scores, (a) item difficulty statistics 
become confounded by valid differences between groups in the ability that a test 
measures and (b) correlational indexes of item discrimination are affected by the 
differences in variability within the groups being compared. Because of these 
complicating factors, traditional item analysis statistics have not proved very 
helpful in detecting differential item functioning. 


Differential Item Functioning 

The proper assessment and study of DIF requires specialized methods and a 
number of them have been proposed. One of the most commonly used is the 
Mantel-Haenszel technique (Holland & Thayer, 1988) which expands on tradi- 
tional item analytic procedures. In this type of analysis each of the groups in ques- 
tion (e.g., majority and minority groups) is divided into subgroups based on total 
test score, and item performance is assessed across comparable subgroups. Al- 
though this method is more refined than the simple comparison of item analysis 
statistics across groups, the Mantel-Haenszel procedure still relies on an internal 
criterion (total score) that may be insensitive to differences in item functioning 
across groups, and its ability to detect DIF is substantially dependent on the use 
of very large groups (Mazor, Clauser, & Hambleton, 1992). 

Item response theory provides a much better foundation for investigating 
DIF than classical test theory methods. In order to establish whether individuals 
from different groups with equal levels of a latent trait perform differently on an 
item, it is necessary to locate persons from two or more groups on a common 
scale of ability. The IRT procedures for accomplishing this goal start by identify- 
ing a set of anchor items that show no DIF across the groups of interest. Once 
this is done, additional items can be evaluated for DIF by comparing the esti- 
mates of item parameters and the ICCs obtained separately for each group. If the 
parameters and ICCs derived from two groups for a given item are substantially 
the same, it may be safely inferred that the item functions equally well for both 
groups. Not surprisingly, IRT procedures are becoming the methods of choice 
for detecting DIF (see Embretson & Reise, 2000, chap. 10, for additional details). 


Applications of Item Response Theory 

As noted in Chapter 3, the use of IRT methods in test development and item cal- 
ibration does not preclude the normative or criterion-referenced interpretation 
of test scores. In fact, because of its more refined methods for calibrating test 
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items and for assessing measurement error, IRT can enhance the interpretation 
of test scores. Although IRT cannot provide solutions to all psychological mea- 
surement problems, it has already helped to bring about a more disciplined and 
objective approach to test development in the areas in which it has been applied. 

At present, IRT methods are being applied most extensively in developing 
computerized adaptive tests used in large-scale testing programs, such as the SAT 
and the ASVAB. Development of tests of this type requires input from individu- 
als with considerable technical expertise in mathematics and computer program- 
ming in addition to knowledge of the content area covered by the tests. More lim- 
ited applications of IRT methods have been in use for some time. For instance, the 
assessment of item difficulty parameters through IRT methods has become fairly 
common in the development of ability and achievement batteries, such as the Dif- 
ferential Ability Scales, the Wechsler scales, the Wide Range Achievement tests, 
and the Woodcock tests. Item-response-theory models are also being used in- 
creasingly in the assessment of DIF in cognitive tests. Although IRT methods hold 
promise in the field of personality testing, their application in this area has been 
much more limited than in ability testing (Embretson & Reise, 2000, chap. 12). 


Explorations in Item Development and Scoring 


The revolution in computer technology, along with the rapid pace of develop- 
ments in the theory and methodology of psychological science, permit an almost 
boundless exploration of innovative techniques that can be applied to measure- 
ment problems. In concluding this chapter, two promising applications of com- 
puter technology to the realm of test items are presented. 


Recent Developments in Item Generation 

The task decomposition and test protocol analysis methods ushered in by cogni- 
tive psychology have led to significant advances in exploring and clarifying the 
processes, strategies, and knowledge stores involved in the performance of test 
items (see Chapter 5). In fact, since the 1980s, these advances—along with con- 
comitant strides in IRT and technology—have been applied in computerized test 
item generation for the development of ability and achievement tests. 'This 
methodology is still in its infancy, comparatively speaking, because the specifica- 
tions needed to create rules whereby computer programs can generate test items 
must be considerably more detailed than they are for traditional methods of gen- 
erating items and require a higher level of theoretical grounding. Nevertheless, 
computerized item generation has already been implemented in developing tools 
for areas—such as mathematics assessment and aptitude testing in aviation—in 
which (a) cognitive models of performance exist, (b) constructs to be examined 
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can be represented in terms of a logical syntax, and (c) item difficulty can be 
gauged through objective referents. Undoubtedly, techniques for item generation 
will continue to be actively pursued by researchers because of the many advan- 
tages they present in terms of efficiency and economy, as well as for their poten- 
tial in pedagogical applications (Irvine & Kyllonen, 2002). 


Automated Essay Scoring 

A significant innovation in the effort to standardize the scoring of essays is the 
development of computer technology for automated essay scoring (AES). The en- 
deavor to evaluate written prose by means of computer software has been in pro- 
gress for the past few decades and, like computerized item generation, has also 
been facilitated by advances in cognitive psychology and computational science. 
Though still in its early stages, AES shows great promise not only as a means of 
increasing the reliability and validity of scores but also as an instructional tool 
(Shermis & Burstein, 2003). 


elm TEST YOURSELF AA 


|. The procedures involved in item analysis pertain primarily to test 
a) developers. 

b) users. 

c) takers. 

d) administrators. 

2. Qualitative item analysis typically takes place 
a) after a test is standardized. 

b) at the same that a test is cross-validated. 

c) after the item pool is generated. 

d) just before a test is published. 

3. Forced-choice items belong to the category of 
a) constructed-response items. 

b) selected-response items. 

c) neither a nor b. 


4. Which of the following is not one of the advantages of selected-response 
over constructed-response items? Selected-response items 


a) are less prone to scoring errors. 


b) make more efficient use of testing time. 


c) are easier to quantify. 


d) are easier to prepare. 
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5. Item discrimination indexes are statistics primarily used to assess item 
(a) validity. 
(b) fairness. 
(c) reliability. 
(d) difficulty. 


6. If one wished to produce a test that would result in maximum differen- 
tiation among test takers, one would aim for an average difficulty level 
(p value) of 


(a) 1.00. 
(b) 0.75. 
(c) 0.50. 
(d) 0. 
7. The Pearson r is the correlation coefficient most commonly used to cal- 


culate indexes of the relationship between item performance and crite- 
rion standing. True or False? 


8. For a pure speed test, the customary indexes of item difficulty and dis- 
crimination are they are for a pure power test. 


a) less appropriate than 
b) more appropriate than 
c) just as appropriate as 


9. Which of the following is not one of the basic objectives toward which 
item response theory (IRT) is geared? 


a) To provide maximum information about the trait levels of examinees 


b) To give examinees items that are tailored to their trait levels 
c) To increase the number of items included in a test 


d) To minimize measurement error 


10. Item response theory, so far, has been applied least extensively in the 
area of testing. 


a) achievement 
b) aptitude 
c) cognitive 


d) personality 


Answers: |. a; 2. c; 3. b; 4. d; 5. a; 6. c; 7. False; 8. a; 9. c; 10. d. 


Seven 


ESSENTIALS OF TEST USE 


tior to their use, tests can be evaluated only from a scientific and technical 
Presson by people who have the necessary expertise in test develop- 
ment, in psychometric principles, and in the aspects of behavior that the 
tests attempt to evaluate. After testing is implemented—through the processes 
of selection, administration, and scoring—test results have to be evaluated, in- 
terpreted, and communicated in a manner appropriate to the purpose for which 
they are to be employed by professionals who have knowledge of the context in 
which the testing takes place as well as of the technical aspects and psychological 
issues involved in a given situation. Ata minimum, evaluating the applications of 
psychological and educational tests involves considerations pertaining to the skill 
with which the test user employs these instruments and to their suitability for the 
test takers with whom they are used. In a larger sense, to the extent that testing 
acquires practical significance in the lives of individuals, test use also needs to be 
evaluated in light of societal values and political priorities. In this wider context, 
testing can become highly controversial and test users must rely on their personal 
and professional codes of ethics to determine whether they ate willing to lend 
their expertise to a particular use of tests. 

Proper testing practices are regulated by the ethical principles and standards 
promulgated by each of the professions that make use of psychological and ed- 
ucational tests (e.g., American Counseling Association, 1995; AERA, APA, 
NCME, 1999; APA, 2002; National Association of School Psychologists, 2000). 
In the past few decades, increasing concerns about the possibility of test misuse 
have led these professions and the organizations involved with testing to become 
engaged in preparing and disseminating information on test user qualifications. 
One of the major efforts in this direction was spearheaded by the APA and re- 
sulted in the publication of an explicit set of guidelines to inform all concerned 
patties of the qualifications that the APA considers important for the competent 
and responsible use of tests (Turner, DeMers, Fox, & Reed, 2001). Rapid Refer- 
ence 7.1 provides a brief summary of these guidelines. In addition, the Testing Stan- 
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Qualifications for Users of Psychological Tests 


As discussed in Chapter |, no formal set of credentials, whether by education, 

licensure, or certification, can ensure competence in the use of a particular test 
in a given situation. 
Rather, the qualifications of users of psychological tests are based on two main 
factors: 


|. Their knowledge and skills in 


e psychometric principles and statistics; 
e selection of tests in light of their technical qualities, the purpose for which 
they will be used, and the characteristics of examinees; 


e procedures for administering and scoring tests, as well as for interpreting, 
reporting, and safeguarding their results; and 
e all matters relevant to the context and purpose of the test use. 
2. The extent to which test users have received appropriate supervised experi- 
ence in all aspects of the knowledge and skills pertinent to the intended use of 
a specific test. 
For further information, see Turner, S. M., DeMers, S. T., Fox, H. R., & Reed, G. M. 
(2001). APA's guidelines for test user qualifications: An executive summary. Ameri- 
can Psychologist, 56, 1099-1 | 13. 


dards (AERA, APA, NCME, 1999) include a chapter that outlines the responsi- 
bilities of test users, as well as separate chapters devoted to issues related to (a) 
fairness in testing and test use, (b) testing individuals of diverse linguistic back- 
grounds, (c) testing individuals with disabilities, (d) psychological testing and as- 
sessment, (e) educational testing and assessment, (f) testing in employment and 
credentialing, and (g) testing in program evaluation and public policy. 
Documents that provide additional guidance in the use of tests as part of high- 
stakes decision making for students (U.S. Department of Education, Office for 
Civil Rights, 2000) and principles for the validation and use of personnel selec- 
tion procedures (Society for Industrial and Organizational Psychology [SIOP], 
2003) are also available and worth consulting prior to undertaking endeavors in 
those areas. A particularly helpful feature of the US. Department of Education 
document is that, besides summarizing the basic principles of sound testing prac- 
tice in the field of educational measurement, it also discusses federal legal re- 
quirements that apply to the nondisctiminatory use of tests for high-stakes pur- 
poses. The SIOP document, while not intended to interpret statutes, regulations, 
ot case law regarding employment decisions, provides principles for the applica- 


256 ESSENTIALS OF PSYCHOLOGICAL TESTING ) 


tion and use of selection procedures that can inform and guide the parties respon- 
sible for authorizing and implementing such procedures in evaluating their ade- 
quacy and appropriateness. 

In recent years, the professions most closely associated with psychological 
testing—which are represented in the Joint Committee on Testing Practices 
(JCTP)—have increasingly come to recognize that test takers also need to be in- 
formed of their rights and responsibilities in the testing process. To that end, they 
have published a document whose sole purpose is to provide such information 
(JCTP, 1998). Rapid Reference 7.2 lists some of the most important rights and 
responsibilities that are outlined and described in this document, which is avail- 
able in its entirety in the Testing and Assessment section of the APA’s Web site 
(http://www.apa.org/science/ttrr.html). 

The application of tests for inappropriate purposes or in inappropriate ways 
by users who lack the proper training and qualifications invariably results in test 


= fid Reference [2 


Rights and Responsibilities of Test Takers 


The document on Rights and Responsibilities of Test Takers: Guidelines and Expecta- 
tions (JCTP 1998) can be freely reproduced and disseminated. Although it does 
not have the force of law—and state and federal laws supercede the rights and 
responsibilities stated therein—professionals involved in the testing process have 
the responsibility to ensure that test takers are made aware of the information 
contained in the document. 


Some of the important rights of test takers include the following: 


e The right to receive an explanation prior to testing about (a) the purposes for 
testing, (b) the tests to be used, (c) whether the test results will be reported to 
them or to others, and (d) the planned uses of the results. If test takers have a 
disability or difficulty comprehending the language of the test, they have the 
right to inquire and learn about possible testing accommodations. 


e The right to know if a test is optional and to learn of the consequences of tak- 
ing or not taking the test, fully completing the test, or canceling the scores. 


e The right to receive an explanation of test results within a reasonable time and 
in commonly understood terms. 


e The right to have test results kept confidential to the extent allowed by law. 
Some of the important responsibilities of test takers include the following: 


e The responsibility to read and/or listen to their rights and responsibilities. 


e The responsibility to ask questions prior to testing about why the test is being 
given, how it will be given, what they will be asked to do, and what will be done 
with the results. 
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DON’T FORGET 


e Even the most carefully developed and psychometrically sound instrument is subject 
to misuse. 


e According to the Testing Standards (AERA, APA, NCME, 1999, chap. ||), the 
responsibility for appropriate test use and sound interpretation of test scores 
rests primarily on the test user 


e Test misuse can occur at every step of the testing process, starting with the in- 
appropriate selection of instruments either for the purposes to which they are 
applied or for the individuals to whom they are administered. Errors in adminis- 
tration or scoring and in the interpretation or reporting of test results may 
compound the problem of test misuse. 


e Whenever the possibility of using tests is contemplated, the best way to pre- 
vent their misuse is to ensure at the outset that the individuals involved in 
every facet of test use have the qualifications and competence necessary to 
fulfill their roles in the testing process. 


misuse. This chapter deals with some of the essential considerations that must be 
taken into account in using psychological tests, including issues related to test se- 
lection, administration, and scoring, as well as to the interpretation and reporting 
of test scores. Since much of the information in former chapters is relevant to the 
fundamental issues involved in responsible test use, readers may want to refer to 
earlier parts of this volume when topics discussed previously are alluded to in the 
present chapter. To the extent that test users and test takers are aware of what 
constitutes sound testing practice, the possibility of test misuse diminishes and 
the potential benefits inherent in the use of tests are more likely to be realized. 


ESSENTIALS OF TEST SELECTION 


As discussed in Chapter 1, psychological tests are used primarily to help in mak- 
ing decisions about people in educational, employment, clinical, forensic, and 
vocational counseling settings. In addition, tests are also frequently used in psy- 
chological research and have recently begun to be applied in the process of 
psychotherapy for purposes of personal development, increased self- 
understanding, or both (Finn & Tonsager, 1997). The consequences of test mis- 
use in these various applications differ widely in terms of their potential impact 
on individual test takers. Naturally, the amount of care devoted to each step of the 
testing process—from test selection on—must take into account and be pro- 
portional to the impact that testing is likely to have. Although most of the ensu- 
ing material is also applicable to the use of tests for research or therapeutic pur- 
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CAUTION 


Of all the arenas in which psychological tests are used, perhaps none is more con- 


pro 


LeS 


We 


An 
tes 


tentious than forensic applications. Due to the adversarial nature of most legal 


ceedings, professionals who agree to appear as experts in trials that involve 


evidence derived from psychological testing can expect to have the bases of their 
timony questioned and challenged at every opportunity. It stands to reason 
that in such situations, so-called experts who are ill prepared to defend their test- 
based testimony can easily be humiliated and embarrassed by lawyers who are 


| prepared to attack it. 
instructive way to become acquainted with the potential pitfalls of misusing 


ts in the legal arena, as well as in other contexts, is by consulting the multivol- 


ume work on Coping With Psychiatric and Psychological Testimony, prepared by Jay 


ZS 


in in collaboration with others and addressed primarily to lawyers. The fifth 


and 
Pre 


latest edition of this work was published in 1995 by Law and Psychology 
ss of Los Angeles, CA. It has since been augmented by supplements issued in 


1997 and in 2000. Of the three volumes in the main work, volume 2—which is 
devoted mainly to challenging testimony derived from psychological tests of vari- 


ous 


kinds—is the most pertinent to the topics considered in this chapter. 


poses, the point of departure for the discussion of most topics in this chapter is 
the assumption that tests are used primarily for making decisions about people. 


To Use or Not to Use (Tests), That Is the (First) Question 


Regardless of the purposes for which psychological testing is intended, the first is- 
sue to be decided is whether testing is needed. In order to resolve this issue, 
prospective test users should engage in a cost-benefit analysis—-similar to the one 
suggested by Goldman (1971)—and explicitly consider the following questions: 


1. 
2. 
3. 


If 


What kind of information do I seek to gain from testing? 

How will this information be used? 

How much, if any, of the information I seek is already available from 
other sources? 

What other tools might be used to gather the information I seek? 
What are the advantages of using tests instead of, or in addition to, 
other sources of information? 

What are the disadvantages or the costs in time, effort, and money of 
using tests instead of, or in addition to, other sources of information? 


the rationale for test use and the expected application of test results are not 


explicit at the outset, test scores are not likely to be of much use or, worse still, 
they are likely to be misused. If the information that is needed is already available 
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from other sources or can be obtained through other means, testing will probably 
be superfluous, unless its purpose is to confirm what is already known or to 
gather additional support for it. Finally, if the advantages or benefits to be gained 
from test use do not outweigh its cost or disadvantages, including any potential 
harm that might accrue from test use, testing is obviously not advisable. Rapid 
Reference 7.3 lists some of the main reasons why and circumstances in which 
testing may be inadvisable. 


Two Important Reasons for Using Tests 

Because testing occurs in so many different settings and is used for so many dif- 
ferent purposes, it is difficult to discuss its impact in the abstract. However, in 
whatever context psychological tests are employed—provided that they have 
been carefully developed and that adequate documentation of their psychomet- 
tic value for a given purpose is available—when they are used judiciously, they 
have distinct advantages over other methods of gathering information about 
people. The most significant advantages that psychological tests offer pertain to 
their characteristic efficiency and objectivity. 


= fgit Reference L3 


Top 10 Reasons for Not Using Tests 


There are many reasons why, and many situations in which, the use of psychologi- 
cal tests is not advisable; the following list merely presents the most salient ones. 
With few exceptions, psychological tests should not be used when any of these 
circumstances apply: 


|. The purpose of testing is unknown or unclear to the test user 


2. The test user is not completely familiar with all of the necessary test docu- 
mentation and trained on the procedures related to the test. 


3. The test user does not know where the test results will go, or how they will 
be used, or cannot safeguard their use. 


4. The information that is sought from testing is already available, or can be 
gathered more efficiently, through other sources. 


The test taker is not willing or able to cooperate with the testing. 
The test taker is likely to incur some harm due to the testing process itself. 


The environmental setting and conditions for the testing are inadequate. 


OND 


The test format or materials are inappropriate in light of the test taker's age, 
sex, cultural or linguistic background, disability status, or any other condition 
that might invalidate test data. 


9. The test norms are outdated, inadequate, or inapplicable for the test taker. 


10. The documentation on the reliability or validity of test scores is inadequate. 
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° Efficiency. Many of the questions that are addressed through psychologi- 
cal testing could be answered by other methods, provided that the indi- 
viduals who seek the required information have the time and resources 
necessary to gather it. For instance, most people do not require the use 
of tests to sketch a psychological profile of those with whom they have 
had extensive and continuous personal contact. By observing the be- 
havior of individuals and interacting with them in a variety of situations 
over a long period of time, ample data can be gathered from which to 
draw conclusions about their skills and personal attributes and even 
about how they are likely to behave in the future. Psychological testing 
affords those who are in a position to evaluate and make decisions 
about people—but do not have the opportunity for prolonged obser- 
vation and interaction with them—the tools for gathering the informa- 
tion they require in a timely and cost-effective manner. 

° Objectivity. Even when there is extensive opportunity for observing and 
interacting with people, the data we can gather through informal obser- 
vations and interactions may be of questionable or limited value. Ob- 
servational data obtained in an unsystematic fashion can easily lead to 
judgments that ate inaccurate on account of the subjectivity of ob- 
servers, including variations in what they observe and remember, as well 
as how well they observe, report, and evaluate their observations. Need- 
less to say, this subjectivity comes into play most particularly when ob- 
servers ate not impartial, as is the case when they are friends, relatives, 
ot associates of the person who is observed. Even when observers are 
keen and able to maintain detachment with regard to the subjects of 
their observation, the raw data gathered from naturalistic observations 
and interactions cannot be appropriately evaluated and interpreted 
without some standardized frame of reference against which to com- 
pare it. As discussed in earlier chapters, the meaning and value that can 
be derived from the behavior samples obtained through standardized 
tests depend almost entirely on the normative or criterion-based frames 
of reference that are available for comparisons and on the accumulated 
data on test score reliability and validity. 


Test Utility 


Whenever the use of tests for making decisions about people is contemplated, the 
potential utility of their use should be considered and analyzed. The concept of 
utility, introduced briefly in Chapter 5 as a consideration that is closely related to 
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validity, refers to an appraisal of the subjective desirability of an outcome or 
event. Utility is an aspect of the much larger topic of decision theory, a widely 
used approach to gathering and analyzing information, usually in mathematical 
form, in order to devise rational strategies for making decisions (see, e.g., Bell, 
Raiffa, & Tversky, 1988). Estimating the value of correct decisions versus the cost 
of incorrect decisions is invariably a complicated matter, especially when the de- 
cisions to be made will affect the lives of human beings in significant ways. Deci- 
sion theory concerns itself with the development and analysis of possible strate- 
gies for decision-making, using available information to estimate their utility by 
calculating the costs and benefits of various alternative outcomes in quantita- 
tive—usually monetary—terms. Naturally, in order to estimate the utility of an 
outcome, a subjective point of view has to be adopted and that point of view typ- 
ically stems from the values and priorities of the decision makers. 

For example, in the field of employment selection there are two possible deci- 
sions (hiring vs. not hiring) that can be made. Assuming that the criterion of job 
performance can be dichotomized into the categories of success or failure, these 
decisions have four possible outcomes: (a) valid acceptances, (b) valid rejections, 
(c) false acceptances, and (d) false rejections. Valid acceptances are the most de- 
sirable of all outcomes; not only do they not pose any hazards for either the em- 
ployer or the applicant who is hired, but they in fact provide benefits to both. 
False acceptances, on the other hand, pose some risks to the employer (e.g,, lost 
revenue or liability incurred due to the employees’ incompetence, wasted time 
and effort in hiring and training activities, increased turnover, etc.) but usually not 
to the employee. Valid rejections are for the most part advantageous to the em- 
ployer but most likely not so to the applicants who are rejected. False rejections 
typically do not present significant risks to the employer, unless the decisions are 
contested, but they may harm the applicants who are falsely rejected, possibly in 
significant ways. 

Clearly, the utility of outcomes differs for the various parties affected by a de- 
cision. Furthermore, the potential consequences of decisions to all parties in- 
volved can never be fully anticipated, let alone quantified. Still, the notion of ex- 
plicitly trying to anticipate and assess the probable benefits and risks inherent in 
using tests—as opposed to other tools—is at the heart of responsible test use. 
To that end, decision theory provides some basic concepts that may be used in 
considering whether and to what extent tests can contribute to improved out- 
comes by increasing the number of correct decisions and minimizing the number 
of incorrect decisions. These concepts are especially suitable for organizations 
seeking to determine possible gains in the accuracy of selection decisions that 
may be achieved by implementing a decision strategy that includes the use of test 
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scores (see, e.g., Kobrin, Camara, & Milewski, 2002). The following are the es- 
sential items of information needed in order to estimate such gains: 


* Validity data: Test scores can improve decision making only to the extent 
that they have demonstrable validity for assessing test takers’ standing, 
ot predicting their performance, on a criterion; all other things being 
equal, the higher the validity coefficient of test scores is, the greater the 
accuracy of criterion estimates and predictions will be (see Chapter 5). 

° Base rate data: The concept of a base rate refers to the starting point from 
which potential gains in accuracy can be calculated, that is to say, the es- 
tablished probability of events in a population prior to the introduction 
of any novel procedure, such as testing. For example, if the proportion 
of students who graduate from a program of study is .60, or 60%, of 
those who were admitted into the program, the base rate is .60. In em- 
ployment selection decisions, the base rate is the proportion of hired 
applicants for a given position who turn out to be successful at their 
jobs. For the purpose of calculating the incremental validity ot improve- 
ment contributed by a test in selection decisions, the base rate refers to 
the proportion of correct decisions that are made without the use of 
test scores. All other things being equal, the potential gains in accuracy 
of selection decisions are greatest when the base rates are close to .50. 
Base rates close to the extremes (e.g., .10 or .90) indicate that accurate 
selection is either very difficult or very easy under existing circum- 
stances. In such cases, using test scores as the basis for selection—even 
if they have a relatively high degree of validity—may not increase the 
accuracy of decisions and might even lower it. 

e Selection ratios: Another constraint on the potential contribution test 
scores might make to the improvement of selection decisions stems 
from the se/ection ratio, which is the ratio that results when the number of 
positions available is divided by the number of applicants for those po- 
sitions. If 5 positions are available and 500 people apply for them, the 
selection ratio is 5 + 500 or 1%, whereas if only 25 people apply, the se- 
lection ratio is 5 + 25 or 20%. Naturally, smaller selection ratios afford 
decision makers the opportunity to be more selective than do larger se- 
lection ratios. As an extreme example, if three positions are open and 
three people apply, the selection ratio is 100% and the employer has no 
latitude for choice if the organization needs to have those workers. In 
such a situation, the incremental validity that test scores can con- 
tribute—regardless of how predictively valid they may be—is nil. On 
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the other hand, when the selection ratio is very small, even a test whose 
scores have only a moderate degree of predictive validity may help to 
increase the rate of accurate decisions. 


As the foregoing discussion makes clear, the degree of improvement in the ac- 
curacy of selection decisions that can be gained through the use of test scores in 
employment and education settings depends on a combination of the base rates 
and selection ratios in a given situation, as well as on the validity of test scores. 
The interactive effects of these three variables on selection accuracy has been ap- 
preciated for a long time; in fact, as far back as 1939, Taylor and Russell published 
a set of tables that display the expected proportion of successes (valid accep- 
tances) that can be expected for various combinations of base rates, selection ra- 
tios, and validity coefficients. The Taylor-Russell tables, which predate the advent 
of decision theory, provide a basic amount of information that can be used in 
evaluating the possibility of using a given test within a certain context, but they 
do not evaluate all possible outcomes of a decision nor do they address all of the 
factors pertinent to selection decisions. 

Since the Taylor-Russell tables were published, industrial-organizational psy- 
chologists have advanced well beyond them and developed additional ways to 
evaluate the effects of test use, incorporating decision theory concepts and mod- 
els. Refinements include the estimation of the effects of test use on outcomes 
other than valid acceptances, such as false rejections, and on outcomes gauged 
through continuous or graduated criteria that may be mote realistic for decision- 
making applications than the simple dichotomy of success or failure embodied in 
the Taylor-Russell tables. Additional aspects of decision making, such as the use 
of multiple predictors in various combinations and the evaluation of strategies for 
selection, classification, and placement of personnel in terms of their utility from 
various points of view (e.g, increasing productivity, decreasing turnover, mini- 
mizing training costs, etc.) have also been investigated in relation to test use in hu- 
man resource management (see, e.g., Boudreau, 1991; Schmidt & Hunter, 1998). 
For a detailed description of one of the most thorough applications of many de- 
cision theory concepts to the development and evaluation of a battery of tests 
and other tools for personnel selection and classification, interested readers are 
referred to J. P. Campbell and Knapp’s (2001) account of the U.S. Army Project 
A (see Chapter 5 for more information on this topic). 


Test Utility in Clinical Decision Making 

Some decision theory concepts may be applied to maximize the efficiency of test- 
ing and assessment in areas other than personnel selection. In clinical psychology, 
for instance, base rates refer to the frequencies with which pathological conditions, 
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such as depression, occur in a given population. Provided that the required in- 
formation is available, knowledge of base rates can help in evaluating the possible 
utility of test use for diagnostic or predictive purposes within a given population. 
As is the case in selection decisions, the contribution that tests can make to im- 
proving predictive or diagnostic accuracy is greatest when base rates are close to 
50. When a condition to be diagnosed is either pervasive or very rare within a 
population, that is, if the base rates are either very high or very low, the improve- 
ment in diagnostic accuracy that can be achieved through the use of diagnostic 
tests is lower than when the condition occurs with moderate frequency. The rea- 
son for this is the same as in selection decisions, namely, that base rates have a lim- 
iting effect on the accuracy of decisions. With regard to diagnoses, if the base rate 
of a condition within a population is very high (e.g., .80), the probability of a false 
positive finding—that is, of diagnosing the condition when it is not present—is low 
(.20) even if diagnostic decisions are made on a random basis. Similarly, if the base 
rate is extremely small, the probability of a false negative finding—that is, of not de- 
tecting the condition when it is present—is also low. In such situations, the cost 
of using test scores to identify the presence of a condition within an entire group 
or population may not be justified by the relatively modest gains in accuracy that 
the test scores could contribute. For an excellent discussion of the implications 
of base rates in clinical practice, the reader is referred to Finn and Kamphuis’s 
(1995) chapter on that topic. 

It should be noted that in clinical settings, as in other fields of assessment prac- 
tice, decisions are usually made on an individual basis using data from a variety of 
tools, including the specialized knowledge and judgment of the assessor. A thor- 
ough review of medical records, including laboratory test results and findings 
from the various imaging techniques currently available, in order to explore or 
tule out physiological factors (e.g., drug use or abuse, neurological or endocrine 
disorders, etc.) that may be causing or contributing to psychiatric symptoms is an 
indispensable part of psychiatric and neuropsychological evaluations in clinical 
settings. Therefore, the contribution made by psychological test results in such 
settings cannot be evaluated in isolation. In addition, the potential utility of indi- 
vidual diagnostic decisions based only on test results is mitigated by the low base 
rates of most mental disorders in the general population (O’Leary & Norcross, 
1998); by the relatively small validity coefficients of many of the tests used for 
such purposes; and by the evolving nature of psychiatric nosology and diagnos- 
tic criteria, especially with regard to personality disorders (see, e.g., Livesley, 
2001). Furthermore, contextual issues related to the reasons why diagnostic as- 
sessment is undertaken can significantly alter the relative ease or difficulty of 
achieving accurate results from testing and other assessment tools. In view of all 
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these factors, itis not surprising that the utility of tests in clinical decision-making 
is a subject of frequent debate within the field of psychology. Rapid Reference 7.4 
lists a few noteworthy contributions to the various sides of this debate. 


Other Assessment Tools 


There are, of course, many avenues that can be used instead of or in addition to 
tests in order to gather information for evaluating and making decisions about 
people. Case history or biographical data, interviewing, systematic and naturalis- 
tic observation, academic and employment records, as well as references from 
teachers and supervisors, are among the most frequently used tools in the assess- 
ment of individuals. Each of these sources of information, singly or in combina- 
tion, can provide valuable data and contribute to successful decision-making. In 
fact, several of these assessment tools (e.g., interviewing, ratings) can be stan- 
dardized and evaluated in terms of their reliability and validity, as well as with re- 
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Are Psychological Tests Useful in Clinical Decision-Making? 


This question is a subject of almost as much debate as the question of whether 
psychotherapy is effective in treating mental disorders. Few would answer either 
one with an unqualified yes or no. Most practitioners who use tests in clinical, 
counseling, and forensic settings believe fervently in their value, whereas often the 
opposite is true for nonpractitioners. For an assortment of opinions on this topic, 
readers might consult the following sources: 


e Eisman, E.J., Dies, R. R, Finn, S. E., Eyde, L. D., Kay, G. G., Kubiszyn, T. W., Meyer, 
G. J., & Moreland, K. (2000). Problems and limitations in the use of psychological 
assessment in the contemporary health care delivery system. Professional Psy- 
chology: Research and Practice, 31, 131—140. 


e Hummel, T. J. (1999). The usefulness of tests in clinical decisions. In J. W. Lichten- 
berg & R. K. Goodyear (Eds.), Scientist-practitioner perspectives on test interpreta- 
tion (pp. 59-1 12). Boston: Allyn & Bacon. 


e Kubiszyn, T. W., Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., 
Dies, R. R., & Eisman, E. J. (2000). Empirical support for psychological assess- 
ment in clinical health care settings. Professional Psychology: Research and Prac- 
aesa l, || ISS HO, 


e Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, 
E. J., Kubiszyn, T. W., & Reed, G. M. (2001). Psychological testing and psychologi- 
cal assessment: A review of evidence and issues. American Psychologist, 56, 128— 
165. (See the Comment section of the February 2002 issue of American Psy- 
chologist for responses to Meyer et al.). 
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gard to their utility in decision-making. When claims concerning the validity of 
such tools for a specific purpose are made, their use falls under the purview and 
guidelines of the Testing Standards (AERA, APA, NCME, 1999, pp. 3—4). If the 
data these methods provide are demonstrably reliable and valid, they may suffice 
as a basis for making decisions, either singly or in combination. 


Biodata 

Life-history information, also known as biodata, can be obtained through a num- 
ber of methods, including interviewing, questionnaires, and examination of exist- 
ing records of past behavior, such as academic transcripts, police reports, and so 
on. As stated earlier, examination of medical records is an indispensable aspect of 
any clinical evaluation of symptoms that may be rooted in or affected by neuro- 
logical or metabolic disorders, drug intake, or other possible physical conditions. 
Moreover, in almost any type of assessment, well-documented data about a per- 
son’s past behavior and achievements are among the most valid and reliable 
soutces of evaluative information because they are factual evidence of what an 
individual is capable of doing or accomplishing. For example, with regard to the 
difficult question of predicting violent behavior, the actuarial approach to risk 
assessment provides the fundamental anchor point for predictions. This 
approach—which can be supplemented by an assessment of current status, as well 
as by clinical information and testing—is based on a systematic examination of 
historical data that have demonstrated empirical relationships to dangerousness, 
such as previous violence, substance abuse, employment stability, mental disorder, 
and early maladjustment (Borum, 1996; Monahan & Steadman, 2001). A similar 
example of the value of historical data can be drawn from the realm of employ- 
ment, where it is generally understood that a proven record of past success in an 
occupation is one of the best possible determinants of future success. Thus, if the 
information about a potential employee’s past job performance is current, reliable, 
and highly favorable, most employers would not hesitate to hire the individual on 
that basis. A more structured and formal approach to the utilization of informa- 
tion on a person’s life history is through biometric data derived from standardized 
biographical inventories, which can be validated as predictors of future perfor- 
mance in a variety of contexts. For a comprehensive account of the use of bio- 
graphical information in selection decisions, interested readers should consult the 
Biodata Handbook edited by Stokes, Mumford, and Owens (1994). 


Interview Data 

Interviewing can provide a wealth of information in almost any assessment con- 
text; it affords the opportunity to observe the interviewee’s behavior and to gather 
pertinent life-history data as well as information about the individual’s attitudes, 
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opinions, and values. A face-to-face interview is a very flexible tool and, when 
properly conducted, may prove to be of critical value in making decisions about 
people. In many clinical, forensic, and employment settings, interviewing the indi- 
viduals who are being assessed as well as those who can provide collateral data is 
considered an essential aspect of the assessment process (see, e.g., Hersen & Van 
Hasselt, 1998). However, the reliability and validity of interview data are highly de- 
pendent on the interviewer’s skill and objectivity in gathering, recording, and in- 
terpreting information. To combat the potential weaknesses inherent in interview 
data, current practices in most fields that use interviewing techniques stress either 
the intensive training of interviewers or the use of structured interviews—which 
are really standardized instruments akin to tests in many ways—or both. 


Observation 

Another ubiquitous source of assessment data consists of ratings, checklists, and 
coding systems based on various types of direct behavioral observation or on pre- 
vious relevant contact with the person to be assessed. As with interviewing, the 
reliability and validity of ratings and other data derived from observation can vary 
greatly depending on the rater and on the system that is used for rating or record- 
ing observations. Training that provides raters with a uniform standard by which 
to evaluate performance can improve the quality of observational data. The use 
of standardized rating scales to gather data from informants, such as parents or 
teachers, is a common procedure in the assessment of children and adolescents 
(see, e.g., Kamphaus & Frick, 2002, chaps. 7 and 8). 


Additional Sources of Information 

Many of the alternative sources of information used in the evaluation and assess- 
ment of individuals cannot be properly evaluated for reliability and validity. For 
instance, the value of letters of recommendation and similar references depends 
on the extent to which the persons providing them are willing and able to disclose 
pertinent information in a conscientious and thorough manner. Those who re- 
ceive the references are most often unaware of factors that may impinge on their 
trustworthiness. Similarly, over the past few decades, grade inflation at all levels 
of schooling has eroded the meaning of many academic credentials to the point 
that grade point averages, and even some degrees and diplomas, can no longer be 
taken at face value with regard to the competencies they signify. 


Searching for and Evaluating Tests 


When the use of psychological tests for a specific purpose is contemplated, 
prospective test users are confronted with two major tasks, namely, (a) finding 
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available instruments for the purposes they have in mind and (b) evaluating them 
from that perspective as well as from the point of view of prospective test takers. 
With regard to the first one of these tasks, readers are referred to the section on 
Sources of Information about Tests in Chapter 1 of this book and, particularly, to 
the document on FAQ /Finding Information About Psychological Tests (APA, 2003), 
created and maintained by the Testing and Assessment staff of the APA and avail- 
able on the APA Web site. As mentioned in Chapter 1, this resource is an excel- 
lent point of departure for the person who seeks information about published or 
unpublished psychological tests. The FAQ document lists and summarizes the 
contents of all the major reference works on tests of both types, and provides in- 
formation on how to locate tests and test publishers, how to purchase tests, avail- 
able software and scoring services, as well as additional information on the 
proper use of tests. 

Evaluating instruments that have been identified as being of possible use fora 
specific purpose is a more complex matter. To be sure, the reviews and the liter- 
ature on the tests in question typically provide basic information about the types 
of scores and coverage provided by a test, the purpose and population for which 
it was designed, how it was developed, its standardization procedures, the types 
of items it uses, possible sources of test bias, and findings related to the reliability 
and validity of its scores, as well as practical features of test content and design 
that have a bearing on the ease with which it can be administered and scored. This 


DON’T FORGET 


Lawrence Rudner's (1994) article on “Questions to Ask When Evaluating Tests,” 
available on the Internet at http://edres.org/pare, provides a good starting point 
for test evaluation. 


Most of the earlier chapters in this book contain material relevant to the evalua- 
tion of tests. A brief overview of some of the key aspects to consider in the pro- 
cess of test selection can be found in the following items: 

e Rapid Reference 3.2: Information Needed to Evaluate the Applicability of a 
ormative Sample 


e Rapid Reference 4.7: Reliability Considerations in Test Selection 


e Rapid Reference 5.8: The Relationship Among Questions, Decisions, and Predic- 
tions Requiring Criterion-Related Validation 


e Rapid Reference 5./2:Validation Strategies in Relation to Test Score Interpreta- 
tion 
e Table 5.1: Aspects of Construct Validity and Related Sources of Evidence 


e Rapid Reference 6.6: What Makes the Behavior Samples Gathered Through Test 
tems So Complex? 
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information can help the test user decide if the instrument, such as it is, appears 
to be appropriate for the purpose and the persons for which it is intended or 
whether local norms or additional data on reliability and validity need to be col- 
lected. However, even when a test is deemed to be appropriate from a psycho- 
metric standpoint, the test user is faced with many other practical issues that are 
uniquely pertinent to the test taker’s specific context and situation, to the level of 
qualifications and experience the use of a test requires, to time and financial con- 
straints, and so on. Rapid Reference 7.5 lists some of the variables that need to be 
considered in relation to the specific individuals or groups to whom a test will be 
administered. 

If the investment of time, effort, and financial resources that will be required 
in order to implement the use of a test within a given context seems likely to be 
justified by the information test scores can provide, the next step for prospective 
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Some Variables to Consider in Selecting Tests in Relation to 
Prospective Test Takers 


e Variables related to test medium: The choice of a specific medium of presenta- 
tion for tests items, such as paper-and-pencil versus computer administration 
or oral versus visual presentation, will pose some advantages or disadvantages 
for examinees, depending on their familiarity with the medium, sensory acuity 
in the auditory and visual modes, motor skills, and so forth. Even when a test 
includes some practice items to familiarize test takers with a chosen medium, 
their varying amounts of prior experience with that medium will probably con- 
tinue to affect their test performance. 


e Variables related to test format: Regardless of their content, selected-response 
items tend to require more receptive skills than constructed-response items, 
which involve the use of expressive skills. Since test takers differ in terms of 
those skills, the choice of item format will also affect their performance. Addi- 
tional variables, such as the use of time limits or individual versus group admin- 
istration, will also affect test takers differentially based on their cultural and ex- 
periential backgrounds. 


e Variables related to the language of test items: Whenever language is part of a 
test but not of its essence, the receptive and expressive linguistic skills of the 
test takers may unduly affect scores. Thus, depending on the requirements of a 
specific test, examiners should consider and ascertain whether test takers have 
sufficient levels of vocabulary, reading skills, and writing skills, to be able to un- 
derstand and attempt to perform the tasks required of them. To this end, for in- 
stance, test manuals should and often do include information concerning the 
reading level required to understand test items. 
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test users is to actually obtain the test 


D O N : T F O RG E T and try it out on themselves or on one 


; ; or more individuals who are willing to 
A unique resource for applied prac- 


tice in reading and interpreting the submit to a practice administration. 


material in test manuals is provided in For this purpose, many publishers 
Ann Corwin Silverlake's (1999) book 


Comprehending Test Manuals: A Guide : 
and Workbook, published in Los Ange- struments that include the test man- 


les by Pyrczak. ual and samples of the materials 
needed for administration and scor- 


make available specimen sets of their in- 


ing. Careful study of these materials, 
especially the test manual, is considered by the Testing Standards as a prerequisite 
to sound test use because they provide the documentation needed for test users 
to evaluate the extent to which test results will serve the purposes for which test 
use is intended. Moreover, test documents provided by the publishers of tests 
must specify the qualifications required for administering and interpreting test 
scores accurately and, whenever possible, alert readers about possible misuses of 
a test (AERA, APA, NCME, 1999, chap. 5). Such documents may include tech- 
nical manuals, user’s guides, and other similar materials intended to supplement 
the test manual. 


ESSENTIALS OF TEST ADMINISTRATION AND SCORING 


Unlike many other aspects of test use considered in the present chapter, the es- 
sentials of test administration can easily be summed up in two words: adequate 
preparation. The proper administration of psychological tests requires careful 
preparation of the testing environment, the test taker, and the person who ad- 
ministers the test. 


Preparing the Testing Environment 


The most important principle to follow in preparing the environment in which 
testing will occur is to anticipate and remove any potential sources of distraction. 
Rooms in which testing is conducted should be adequately lit and ventilated, 
should provide appropriate seating and space for test takers, and should be free 
from noises or other stimuli (e.g., food or drink) that might disrupt the test tak- 
ers’ ability to attend to their tasks. To prevent possible interruptions, it is cus- 
tomary to post a sign—which many test publishers freely provide—on the door 
of the examination room to alert passers-by that testing is in progress. 

Beyond securing a suitable testing room, adequate preparation of the testing 
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environment involves following the test manual’s instructions for administration, 
which are geared toward replicating the conditions under which the test was stan- 
dardized as closely as possible. For group testing, these instructions might in- 
clude providing materials necessary for taking the test (e.g, pencils with erasers) 
and making sure that test takers are seated in such a way that they are prevented 
from conversing or looking into each others’ response booklets, enlisting test 
proctors, and so forth. For individual tests, special seating arrangements need to 
be made so that the examiner can present test materials in the proper orientation, 
record responses unobtrusively, and fully observe the test taker’s behavior. 

As a general rule, the presence of anyone other than the examiner and the test 
taker in the room where an individual test administration takes place should not 
be allowed (see, e.g., National Academy of Neuropsychology, 1999). The pres- 
ence of third parties poses the possibility of distracting from or even influencing 
the testing process and introduces an element that is inconsistent with standard- 
ized test administration and an additional, unnecessary risk to test security. There 
may be special circumstances that require the observation of a test administration 
by others—for example, students who are receiving formal training in test ad- 
ministration. Ideally, such observations should be made from a room adjacent to 
the testing room, through a one-way mirror. For test takers who may have diffi- 
culties communicating with the examiner because of their young age or linguistic 
background, or because of a disability, the presence of a parent or an interpreter 
may be required in the testing room. In such situations, as well as in any other case 
in which special accommodations that may have a bearing on the interpretation 
of scores are made, the report of test results should note them. 


Preparing the Test Taker 


There are two distinct aspects of testing that relate to the preparation of test tak- 
ets. The first one is largely within the purview of the examiner and concerns the 
establishment of rapport as well as the proper orientation of the test taker prior 
to administering the test. The second aspect, which is not within the examinet’s 
control, pertains to all the antecedent life experiences that test takers might have 
had that would affect their performance on a particular test. 


Establishing Rapport 

In the context of testing, the term rapport refers to the harmonious relationship 
that should exist between test takers and examiners. In order to maximize the re- 
liability and validity of test results, a friendly atmosphere needs to be established 
from the beginning of the testing session and rapport ideally should range from 
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good to excellent. Absence of rapport can, of course, be attributable to either 
party in the test situation and may stem from inexperience or ineptness on the 
part of the examiner, an unfavorable disposition on the part of the test taker, or 
both. Needless to say, to the extent that rapport is lacking, test performance is 
likely to be deleteriously affected, even to the point where test scores are invali- 
dated. In order to build rapport, the examiner should attempt to engage the in- 
terest and cooperation of test takers in the testing process so that they may react 
to test tasks in an appropriate fashion, by putting forth their best efforts in tests 
of ability and by responding as openly and honestly as possible on instruments 
designed to assess personality. 

The process of rapport-building is more extensive in individual than in group 
testing because individual testing allows examiners to observe the test taker’s be- 
havior closely and continuously and to extend their efforts to maintain rapport 
throughout the testing process. Nevertheless, even in group testing, examiners 
must try to explain the purpose of the test, the testing procedures, and so on (see 
Rapid Reference 7.2) in a friendly manner within the constraints of the directions 
for test administration provided in the test manual, which must be followed in or- 
der to keep conditions uniform. 


Test Preparation From the Test Taker’s Perspective 

It goes without saying that the purpose for which testing takes place, and the con- 
textual aspects of the situation and circumstances in which testing takes place, 
have a significant bearing on test takers’ attitudes and motivation. Among the 
many other cognitive and emotional factors that have a bearing on examinees’ 
predispositions toward testing and on the extent to which they are prepared for a 
test-taking experience, test anxiety and test sophistication are probably the two 
most frequently discussed and investigated in the psychological testing literature. 
A brief overview of each follows. 

Test anxiety. The prospect of being evaluated tends to elicit some degree of appre- 
hension in most test takers, especially when the test scores are to be used as a ba- 
sis for making decisions that will have important consequences for them. For 
some test takers this apprehension is easily dissipated and dealt with—and may 
even improve their levels of performance by heightening their physiological 
arousal. For others, test anxiety becomes a debilitating or even incapacitating fac- 
tor that may have a significant deleterious effect on test performance or prevent 
them from taking tests altogether. Moreover, the reasons why test takers experi- 
ence anxiety before and during the testing process can vary widely as well. Some 
of these reasons may be related to the type of test to be taken (e.g., speed tests, 
math tests, etc.), some may be related to test takers themselves (e.g., expectations 
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of failure based on antecedent experiences), and still others may be a function of 
the context in which testing takes place (e.g., preemployment testing) or of a com- 
bination of variables. Although examiners should be alert to try to reduce the 
level of test takers’ anxiety as part of the process of building rapport, there are also 
measures that test takers themselves can take toward the same end. Rapid Refer- 
ence 7.6 lists resources that may be helpful to test takers and others interested in 
current approaches to the assessment and treatment of test anxiety. 

Test sophistication. Strictly speaking, the variable known as fest sophistication (a.k.a. 
test-taking skills or test wiseness) refers to the extent to which test takers have had 
experience or practice in taking tests. As a general rule, on most types of ability 
tests, having had the experience of taking a particular test or type of test tends to 
be advantageous for the test taker in that it provides practice and may reduce anx- 
iety and bolster confidence. In fact, when individuals are retested with either the 
same or an alternate form of a test of ability, their second scores are almost in- 
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Sources of Information on Test Anxiety 


Test takers who want help in coping with test anxiety will find a wealth of materi- 
als available in bookstores and on the Internet. Examples include the following: 


e Taking the anxiety out of taking tests: A step-by-step guide, by S. Johnson. New 
York: Barnes & Noble Books, 2000. 


e No more test anxiety: Effective steps for taking tests and achieving better grades, 
by E. Newman (available with audio CD). Los Angeles: Learning Skills Publica- 
tions, 1996. 


e The Test Anxiety Scale (Saranson, 1980), which provides a quick way to gauge 
the extent to which one may be prone to experience test anxiety and is avail- 
able free of charge from Learning Skills Publications (at http://www.learning 
skills.com/test.html) and several other Internet sites. 


e Many Web sites sponsored by university counseling centers are accessible by 
searching for “test anxiety” on the Internet; these sites provide tips on study 
habits and other information on coping with test anxiety. 


For those who wish to delve further into the current theories and research on 
test anxiety, the following works are recommended: 


e Sapp, M. (1999). Test anxiety: Applied research, assessment, and treatment (2nd 
ed.). Latham, MD: University Press of America. 


e Spielberger, C. D., & Vagg, P R. (Eds.). (1995). Test anxiety: Theory, assessment, and 
treatment. Washington, DC: Taylor & Francis. 


e Zeidner, M. (1998). Test anxiety: The state of the art. New York: Plenum. 


274 ESSENTIALS OF PSYCHOLOGICAL TESTING J 


variably higher than the first, a phenomenon that is known as the practice effect. 
Naturally, given the enormous range in the kinds of tests and test items that ex- 
ist, a test taker may be very sophisticated and practiced in taking tests of one type 
in a given medium (e.g., multiple-choice paper-and-pencil achievement tests) but 
not at all familiar with tests of other types in other media (e.g., individually ad- 
ministered general ability tests or computer-administered performance tests). By 
the same token, test takers differ greatly in terms of the spectrum of experiences 
they have had that may have made them less or more prepared for a test-taking 
experience. 

Within the testing process itself, the traditional way of coping with variability 
in test sophistication has been to provide explicit instructions and practice items 
prior to the test as part of the standardized administration procedures, in order to 
ascertain that test takers are able to appropriately handle the mechanics of re- 
sponding to test items. Although these practice orientation sessions cannot by 
any means obliterate the differences among test takers, they can at least ensure 
that examinees will be able to manage the test-taking procedures competently. In 
addition, it is always good testing practice for examiners to inquire about the test 
takers’ prior experiences with the actual test or type of test they are about to take 
and to note this information for use in the interpretation of test scores. 

Of course, there are many other avenues that test takers can use to acquire 
skills and practice in test taking. One of the simplest and most efficient methods 
for prospective takers of college and graduate school admission tests is to famil- 
iarize themselves with the items and procedures of such tests by taking the sample 
tests provided by the publishers of these instruments (see, e.g., Rapid Reference 
6.2). Teachers and counselors can also be of help in providing students and clients 
with information and guidance in test-taking skills (see, e.g., Scruggs & Mas- 
tropieri, 1992). The importance that many people place on test preparation is un- 
detscored by the fact that coaching for tests has become a major industry that 
provides test takers with a multiplicity of manuals, software, courses, and tutor- 
ing services geared to a variety of testing programs. Whether and to what extent 
test takers are able to achieve significant score gains through various methods of 
coaching and test preparation is a subject of much debate. An interesting account 
of some of the issues and findings on this topic is provided in the chapter on 
“Gaming the Tests: How Do Coaching and Cheating Affect Test Performance?” 
in Zwick’s (2002) book Fair Game? The Use of Standardized Admissions Tests in Higher 
Education. 

Another aspect of the debate about test preparation pertains to the distinction 
between intensive drilling aimed solely or primarily at raising test scores, on one 
hand, and teaching that addresses the broader objectives of the curriculum, on the 


< ESSENTIALS OF TEST USE 275 


other hand. This issue has been highlighted by the rather controversial practice of 
mandated testing that many states and localities are increasingly instituting for the 
putpose of making public schools accountable for student learning. In this case, as 
in many others, economic and political considerations create conditions in which 
the very purpose of testing may be subverted and the responsibilities of various 
parties in the educational process deflected from the test users to the tests. 


The Problem of Test-Taker Dissimulation 

An entirely different perspective on test takers’ predispositions is presented in sit- 
uations in which the type of decision-making for which a test is used promotes 
dissimulation. Attempts on the part of test takers to present themselves in either 
an unrealistically favorable or unfavorable fashion are not at all uncommon and 
may or may not be conscious. Validity scales designed to detect various types of 
attempts at impression management or response sets such as defensiveness are 
embedded in a number of personality inventories (e.g., the MMPI-2 and the 
MCML-III) and have a long history. In recent years, instruments designed espe- 
cially to evaluate the possibility of malingering on cognitive tests administered in 
the context of forensic and neuropsychological assessment—such as the Validity 
Indicator Profile—have been added to the repertoire of tools available for that 
purpose (see, e.g., R. Rogers, 1997). Undoubtedly, obtaining the full cooperation 
of test takers in the testing process is a crucial matter upon which depends the re- 
liability and accuracy of test results (see Rapid Reference 7.3). 

For a survey of the research on many additional topics that have a bearing on 
the test taker’s outlook on educational and psychological testing, readers might 
wish to consult Nevo and Jager’s (1993) edited volume on this topic. The studies 
presented in this work were conducted by investigators in Germany, Israel, and 
the United States who were seeking to gather feedback on examinees opinions, 
attitudes, and reactions to various aspects of their test-taking experiences with 
the ultimate goal of improving specific tests and testing in general. 


Preparation of the Examiner 


Obtaining Informed Consent 

According to the Ethical Principles of Psychologists and Code of Conduct (APA, 2002), 
prior to the administration of a psychological test or assessment procedure, psy- 
chologists must obtain and document the żnformed consent of test takers either 
orally or in writing. In order to be considered nformed, the consent obtained from 
test takers has to be preceded by a suitable explanation of the nature and purpose 
of the evaluation, as well as information concerning confidentiality limits and 
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how the security of test results will be maintained. Other practical matters that 
might affect test takers in a specific situation (e.g., fees, ability to refuse or dis- 
continue testing, involvement of third parties, etc.) should also be discussed. Test 
takers should be given the opportunity to ask questions they might have about the 
testing process and have them answered. The code of ethics of psychologists 
(APA, 2002) lists a few exceptional circumstances in which the informed consent 
is not required either because it is implied or because the assessment is mandated. 
Special provisions that pertain to such cases, as well as to persons who do not 
have the capacity to consent, are also outlined in the code. Other professions en- 
gaged in the use of psychological testing and assessment instruments have simi- 
lar guidelines for the attainment of informed consent (see, e.g., American Coun- 
seling Association, 1995). 


Importance of Examiner Preparation 

The key role that proper administration and scoring of tests plays in gathering in- 
terpretable data cannot be overemphasized. Standardized procedures for the ad- 
ministration and scoring of a test as specified in its manual provide the foundation 
that allows for the application of a normative or criterion-based frame of reference 
to the interpretation of test scores. To be sure, deviations from or modifications of 
standardized procedures are sometimes inevitable or necessary, as, for instance, 
when the administration of a test is disrupted by some event or when special ac- 
commodations need to be made for test takers with disabilities. In such cases, ex- 
aminers need to document the modifications that were made. Moreover, to the ex- 
tent that there is reason to believe that test score meaning may be affected by 
disruptions or modifications of standardized procedures, the nature of the dis- 
ruptions or modifications should be reported to those who will be interpreting or 
making decisions on the basis of the test scores (AERA, APA, NCME, 1999). 

As far as the actual preparation of examiners is concerned, the person who 
administers a test has to be thoroughly familiar with the purpose and procedures 
of the test, able to establish rapport, and ready to answer test takers’ questions or 
cope with any foreseeable emergency that might arise during testing. In general, 
group test administration does not require any additional training beyond the 
aforementioned. Individual testing, on the other hand, inevitably involves a great 
deal more preparation and usually requires supervised experience. For instance, 
when test questions are presented orally they must be stated verbatim. In order 
for the administration to go smoothly, proceed at a good pace, and allow exam- 
iners the opportunity to attend to and record responses (also verbatim), examin- 
ers need to have committed to memory not only the items and their sequence, but 
also a number of additional rules for such things as timing the presentation of and 
responses to items and the starting and stopping points, as well as for scoring the 
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responses as they are produced. When materials such as puzzle pieces, blocks, 
pictures, and other objects are used in individual testing, they must be presented 
and removed in the exact manner that is called for in the manual. Unless examin- 
ets memorize and practice all test procedures beforehand until they are thor- 
oughly adept at them, the test administration and possibly the test taker’s perfor- 
mance, as well, will be affected in ways that could easily jeopardize the validity of 
the test data. 


Computer-Based Test Administration 

One sure way to bypass the possibility of errors in test administration, as well as 
the likelihood that test performance might be unduly influenced by variables re- 
lated to the examinet’s sex, race, age, appearance, interpersonal style, and other 
such variables, is through computer-based test administration. The advantages of 
this testing medium, with regard to the relative uniformity in the presentation of 
test materials and the precision with which responses can be timed, recorded, and 
scored, are self-evident. In fact, the trend to replace paper-and-pencil testing with 
computerized test administration is well underway, especially in large-scale test- 
ing programs, and is likely to continue to expand as computerization becomes 
more cost effective. Moreover, the development and use of tests—such as the 
Test of Variables of Attention—that by their nature can exist only in computer 
versions, is also progressing rapidly. On the other hand, the administration of in- 
dividual tests of general ability, projective techniques, and many neuropsycho- 
logical tests (provided administration is performed by a skilled examiner) has 
some definite advantages in terms of the qualitative data that can be gathered, es- 
pecially in the context of clinical, neuropsychological, and forensic assessments. 
In addition, prior to instituting a change from traditional methods of administer- 
ing tests to computerized administration, the comparability of results under the 
two sets of conditions—which varies depending on the type of instrument in 
question—needs to be thoroughly investigated (Mead & Drasgow, 1993). For 
more information regarding computerized psychological assessment, including 
the issue of the equivalence of standard versus computerized test administration, 
see Butcher (2003). 


Test Scoring 

It may be recalled that interscorer differences and the associated topic of scorer re- 
liability—discussed in Chapter 4, among the possible sources of measurement er- 
ror—were considered as pertinent only to the scoring of open-ended responses in 
which the subjectivity of the scorer plays a part. When this is the case, test users 
need to ascertain empirically that those who will be scoring such responses—from 
individual or group testing—are trained well enough to achieve results virtually 
identical to those produced by an independent and experienced scorer. As a gen- 
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eral rule, when interscorer reliability coefficients can be computed, they should 
approach +1.00 and should not be much below .90. When scores are expressed in 
some other fashion, the goal should be to get as close as possible to 100% agree- 
ment in the scores assigned by independent, trained scorers. It should be noted 
that the ease with which such high standards of accuracy can be met when the scor- 
ing requires subjective judgment differs greatly across different types of tests. This 
is so because guidelines for scoring some open-ended responses (e.g, essay an- 
swets on achievement tests) tend to be clearer and easier to master than those for 
others (e.g., responses to projective techniques such as the Rorschach). For a dis- 
cussion of issues related to scorer reliability for the Rorschach Comprehensive 
System, see Acklin, McDowell, Verschell, and Chan (2000). 

In tests that are scored objectively—simply by counting responses in various 
categories and performing the computations required to transform raw scores 
into some other numerical form, such as standard scores of various types—scor- 
ing errors need not occur. One way to avoid them is through computer-based test 
administration and scoring, Another is through the use of optical scanners and 
appropriate software, although this requires the careful examination of answer 
sheets for incomplete erasures and other such problems prior to scanning. If ob- 
jective scoring is done by hand, especially when templates that are superimposed 
on answer sheets ate involved, the possibility of clerical errors has to be fore- 
stalled by impressing upon scorers the absolute importance of accuracy and by in- 
stituting procedures to double-check all of the required calculations and score 
transformations. 

The transformation of the raw scores of individually administered tests, such 
as the Wechsler scales, into standard scores typically involves a series of arith- 
metical computations as well as looking up the scaled-score equivalents of raw 
scores in various tables provided in the manuals. To prevent errors from entering 
into scores as a result of carelessness in carrying out these procedures, test pub- 
lishers also offer software that can perform all the required calculations and trans- 
formations once the raw scores obtained by the examiner are (carefully) entered 
into the computer. If test users are not able to avail themselves of this kind of 
scoring aid, good testing practice requites double-checking the accuracy of all 
computations and equivalent scores obtained from tables. 


ESSENTIALS OF TEST SCORE INTERPRETATION 
AND REPORTING 


Most of the preceding material in this book has been aimed at communicating to 
readers the complexities involved in the proper use and application of psycho- 
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logical test data. Test score interpretation and the communication of the infer- 
ences obtained from psychological testing are the culminating points of test use. 
They are also the components of psychological testing through which test takers 
can either derive the greatest benefit, or incur the greatest harm, that can flow 
from test use. Since all the various principles involved in the interpretation and 
reporting of test results cannot be conveyed in a single book—let alone in one 
portion ofa chapter—this section presents a general perspective on these aspects 
of testing and selected references on related topics. 


A Particular Perspective on Test Interpretation 


Psychological test scores supply more ot less reliable quantitative data that con- 
cisely describe the behavior elicited from individuals in response to test stimuli. To 
the extent that tests are carefully selected, administered, and scored, they provide 
information that can be used in a variety of ways, the most basic of which is 
simply to place test takers’ performance within the descriptive normative or 
ctiterion-based categories provided by the frame of reference a test employs (see 
Chapter 3). If they are sufficiently reliable and appropriately interpreted, test 
score data can also be of help in explaining the psychological make-up of indi- 
viduals and in making decisions about them based on the estimates that scores 
provide concerning test takers’ characteristics or their future behavior (see Chap- 
ter 5). However, to arrive at defensible answers to the complex questions that 
psychologists, teachers, and other human services professionals are asked, or ask 
themselves, data from multiple sources are often necessary and informed judg- 
ment must always be exercised. 

Making informed judgments about people requires an appreciation of the 
value and limitations inherent in the frame of reference, reliability, and validity of 
test score data, as well as in the data from all the other sources that might be em- 
ployed in a given case. In addition, it requires knowledge about the context and 
specific areas of human behavior relevant to the issue in question. Furthermore, 
making decisions about people invariably involves value judgments on the part of 
the decision makers and places upon them an ethical responsibility for the con- 
sequences of the decisions that are made. Unfortunately, in actual practice, con- 
textual issues are frequently ignored, value judgments are not acknowledged ex- 
plicitly, and test scores often become the major or even the sole determining 
factor in decision-making. As a result, for reasons of expediency, the weight of re- 
sponsibility for many decisions is unjustifiably transferred from test users and de- 
cision makers to the tests themselves. One way to counteract this problem is to 
understand and appreciate the implications of the difference between psycho- 
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logical testing and assessment, discussed in Chapter 1. This difference is akin to 
the distinction between conducting medical tests, on one hand, and integrating 
their results with the history and presenting symptoms of a patient to produce a 
diagnosis and treatment plan, on the other (Handler & Meyer, 1998). 

The overview of strengths and limitations of testing and other assessment 
tools presented earlier in this chapter attempted to convey the desirability of in- 
tegrating as many soutces of evidence as possible whenever a significant question 
that calls for the assessment of individuals or groups is asked. With regard to the 
interpretation and use of tests, the particular perspective presented here is as fol- 
lows: 


1. Psychological tests can sometimes be the most efficient and objective 
tools available for gathering reliable and valid data about people. 

2. Psychological testing can often be a valuable component in the process 
of assessment of individuals and groups. 

3. Psychological test scores should never be the only source of informa- 
tion on which to base decisions that affect the lives of individuals. 


The particular manner in which the interpretation and reporting of test scores 
should be conducted depends on two interrelated factors: (a) the purpose for 
which the testing was undertaken and (b) the party on whose behalf the testing 
was undertaken. With respect to the latter, three distinct possibilities determine 
how test data are interpreted and communicated. 


° When psychologists use tests on their own behalf (e.g., as instruments in re- 
search), they can interpret and report the grouped test score data of re- 
search participants in whatever manner they deem appropriate to the pur- 
poses of their investigations. Legal requirements and ethical standards for 
reseatch with human participants govern practices in this type of testing 
and include provisions for obtaining the informed consent of participants 
and debriefing them promptly (APA, 2002). With regard to the role of test- 
ing within the research project itself, issues such as the choice of instru- 
ments, the way scores are reported, and the meaning assigned to them must 
be evaluated in light of the goals of the investigation. Scientific research and 
publication are essentially self-policing enterprises that use the mechanism 
of peer review to evaluate the substantive and methodological merits of 
proposed, ongoing, or completed work. Thus, when research that entails 
the use of test data is submitted for review or is published, it will either be 
accepted or rejected, cited or ignored, based—among other things—on 
how well such data were employed. 
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° When psychologists use tests on behalf of their own clients, they ate singularly 
responsible for interpreting test data, integrating them with other sources 
of information, and communicating their findings to their clients in an ap- 
propriate and helpful manner. Whether these assessments are aimed at di- 
agnosis, treatment planning, monitoring progress, or facilitating change, 
the client is the ultimate consumer of the information and the ultimate 
judge of its benefits. In fact, in many of these situations, test score inter- 
pretation, as far as the implications of scores ate concerned, can be a col- 
laborative effort between the psychologist or counselor and the client (Fis- 
cher, 2000). Rapid Reference 7.7 lists resources that provide explicit 
guidance on the use of tests within the context of counseling and clinical 
assessment. 

° When psychologists use tests on behalf of a third party, such as an organiza- 
tion or another professional, they are acting as consultants in an assess- 
ment process initiated by others for their own purposes. Hence, the lines 
of responsibility are not as clear-cut as in the previous instance because the 
test taker who is undergoing assessment is not necessarily the ultimate 
consumer of test data. Nevertheless, as mentioned elsewhere, both from 
an ethical standpoint as well as for protection from possible liability, it be- 
hooves consultants to find out what the purpose of the consultation is, not 
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Resources on the Use of Tests in Counseling and 
Clinical Practice 


Articles and books that provide guidance and examples on the interpretation and 
use of psychological tests in clinical practice abound and their number is constantly 
expanding. The works included in this brief list present just a few of the many pos- 
sible ways to apply testing in counseling and clinical settings. Resources for more 
specialized applications and populations are included in Rapid Reference 7.8. 


e Beutler L. E., & Groth-Marnat, G. (Eds.). (2003). Integrative assessment of adult 
personality (2nd ed.). New York: Guilford. 

e Fischer, C. T. (1994). Individualizing psychological assessment. Hillsdale, NJ: Erl- 
baum. (Original work published 1985) 

e Lowman, R. L. (1991). The clinical practice of career assessment: Interests, abilities, 
and personality. Washington, DC: American Psychological Association. 


e Maruish, M. E. (Ed.). (2004). The use of psychological testing for treatment planning 
and outcome assessment (3rd ed., Vols. 1—3). Mahwah, NJ: Erlbaum. 
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only in terms of what information the third party is seeking, but also how 
it will be used. Only with this knowledge can assessment professionals de- 
termine whether the use of tests is called for, how scores will be reported, 
what other tools may be needed to derive the information that is sought, 
and whether they are able and willing to participate in the consultation 
process. 

The interpretation of test results involves a series of inferences that ate 
made on the basis of the data gathered from (a) the actual behavior samples 
(responses to test items), (b) the aggregation of these samples into one or 
more scores, (c) the available evidence concerning the reliability of the 
obtained scores, (d) the comparison of scores against the normative or 
ctiterion-based frames of reference the test provides, (e) the evaluation of 
scores in light of the quality of internal and external validation data avail- 
able, (f) the specific context and situation in which the testing takes place, 
and (g) the personal characteristics of the individual being assessed. When 
all of these sources of evidence are viewed in conjunction with each other, 
and added to the pertinent information collected through other methods, 
their implications and limitations with regard to a specific assessment ques- 
tion should be clear. Occasionally (e.g., when data from various reliable and 
presumably valid sources are mutually contradictory or when there is rea- 
son to believe that some of the key pieces of evidence—whether derived 
from tests or other sources—are unreliable), the inconclusive nature of 
findings must be reported. In such cases, appropriate recommendations 
might include referral to another professional or additional data collection. 
Rapid Reference 7.8 lists a few of the many resources currently available on 
test interpretation and related topics. 


Communicating Test Results and Assessment Findings 


The most basic guideline to follow in communicating test results is to provide the 
information derived from test scores, including its limitations, in language that 
the recipient can understand. However, the specific manner in which scores are 
reported can vary widely depending on the tests administered, the setting or con- 
text in which testing takes place, the purposes for which the testing was under- 
taken, and the intended recipients of the information. Thus, the appropriate way 
to report the results of psychological testing or the findings of an assessment can- 
not be condensed into a single set of rules suitable for all cases. Nevertheless, is- 
sues pertaining to some of the various modes of communicating test results are 
presented in the ensuing paragraphs. 
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Further Information on Test Interpretation and Assessment 


The primary sources of information on the use and interpretation of specific tests 
and assessment tools are the manuals, guidebooks, and other supporting docu- 
mentation provided by test authors and publishers. In addition, guidelines on test 
interpretation can be found in a great number of books and publications. A few 
examples follow. 

For a general overview of test interpretation: 


e Lichtenberg, J. W., & Goodyear, R. K. (Eds.). (1999). Scientist-practitioner perspec- 
tives on test interpretation. Boston: Allyn & Bacon. 


For issues related to the assessment of children and adolescents: 

mphaus, R. W. (2001). Clinical assessment of child and adolescent intelligence 
(2nd ed.). Boston: Allyn & Bacon. 

e Kamphaus, R. W., & Frick, PJ. (2002). Clinical assessment of child and adolescent 
personality and behavior (2nd ed.). Boston: Allyn & Bacon. 

* Sattler J. M. (2001). Assessment of children: Cognitive applications (4th ed.). San 
Diego, CA: Author. 

* Sattler, J. M. (2002). Assessment of children: Behavioral and clinical applications 
(4th ed.). San Diego, CA: Author. 

For perspectives on test interpretation for diverse populations: 

e Sandoval, J., Frisby, C. L., Geisinger, K. F., Scheuneman, J. D., & Grenier, J. R. 


(1998). Test interpretation and diversity: Achieving equity in assessment. Washing- 
on, DC: American Psychological Association. 


e Ekstrom, R. B., & Smith, D. K. (Eds.). (2002). Assessing individuals with disabilities 
in educational, employment, and counseling settings. Washington, DC: American 
Psychological Association. 


& 


° When test results are communicated directly to test takers in score report cards or 
profiles produced by computers, the organization responsible for the testing pro- 
gram must provide adequate interpretive information. For instance, the 
College Board Web site (at http://www.collegeboard.com/student/ 
testing/sat/scores /understanding.html) has some material pertinent to the 
interpretation of SAT scores under the heading “Understanding Your 
Scores.” Similar aids are available for interpreting scores derived from other 
large-scale testing programs, such as the ACT (at http://www.act.org/aap/ 
scores /under.html). 

© Interpretations of scores derived from computer programs apply decision rules 
based on the clinical experience and judgment of experts, actuarial ap- 
proaches based on statistical associations and correlations between scores 
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and criteria, or both. How these computer-based test interpretations 
(CBTIs) are employed, and by whom, is a subject of continuing discussion 
and debate within the testing profession. Originally meant as aids to clini- 
cians in the interpretation of personality inventories and other diagnostic in- 
struments, CBTIs have proliferated rapidly and have been extended to dif- 
ferent kinds of tests and test uses. One of the problems stemming from 
these developments is that CBTI services are often made commercially 
available to individuals who may have seemingly appropriate credentials— 
such as a license to practice psychology or a medical degree—but who are 
not sufficiently knowledgeable about the test in question. As a consequence, 
the reports generated through CBTI are too often viewed uncritically and 
used as substitutes for individualized assessment that is based on multiple 
data sources and informed by the judgment of a qualified professional. In 
addition, the proprietary nature of the specific rules used in generating such 
reports often prevents a proper evaluation of the validity of their interpreta- 
tions by prospective users. Nevertheless, professionals who avail themselves 
of CBTI services are under the same ethical obligations with regard to com- 
petence in their use as in the use of any other assessment tool. The providers 
of CBTI services, in turn, have an ethical obligation to supply their users 
with information concerning the sources, bases of evidence, and limitations 
of the interpretations they provide (see, e.g., APA, 2002; Moreland, 1991). 

° The traditional means for communicating test results and assessment findings is the 
written psychological report. This individualized approach affords a great deal 
of flexibility in tailoring the communication to the purpose of the assess- 
ment and to the needs of clients and consumers of psychological test data. 
It also helps the assessor organize, clarify, and synthesize information from 
all the available sources and creates a record of the findings of an assess- 
ment that can be consulted in the future. A comprehensive description of 
the various approaches to psychological report writing and their benefits 
and pitfalls, as well as numerous examples of how to and how not to write 
such reports, are provided by Norman Tallent (1993). A briefer, step-by- 
step guide to report writing prepared by Raymond Ownby (1997) is also 
worthy of the attention of readers interested in this topic. Rapid Reference 
7.9 lists a few examples of do’s and don'ts in test interpretation that are es- 
pecially pertinent to psychological report writing. 


Safeguarding Test Data 

The security of test data, whether they consist of individually identifiable records, 
scores, and reports or of test materials themselves (e.g., booklets, forms, ques- 
tions, scoring keys, manuals, etc.), is a primary responsibility of the test users and 
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Do’s and Don’ts in Test Interpretation 


Some Examples of What Test Interpretation Is Not 


e Reporting numerical scores: Whether scores are expressed as percentile ranks, 
IQs, percentage scores, or in some other format, simply listing them fails to 
convey their meaning and implications. 


e Assigning labels: Placing individuals within diagnostic categories (e.g., Mild Mental 
Retardation, or Borderline Personality) or typologies on the bases of their 
scores is not an adequate substitute for interpretation that enhances under- 
standing. 


e Stating findings in terms of trivial generalities: Many statements can apply equally 
well to almost all human beings or to most individuals in certain categories 
(e.g., psychiatric patients, young children, etc.) because of their high base rates 
in those populations. Meehl (1956) proposed the phrase “Barnum effect” to 
characterize the worthless nature of descriptions of this sort. 


What Test Interpretation Should be 


e At a minimum, the interpretation of test scores for consumers of test data 
should include a clear explanation of (a) what the test covers, (b) the meaning 
of scores, (c) the limitations on the precision of scores that derive from mea- 
surement error, (d) common misinterpretations to which particular scores— 
for example, |Qs—may be subject, and (e) the way in which test results may or 
will be used (AERA, APA, NCME, 1999, chap. 5). 


e At its best, test score interpretation adds value to the aggregated behavior 
samples collected with tests by integrating them with all other available data 
and using informed professional judgment to draw useful and ecologically valid 
inferences. 


institutions who control access to such data. In certain cases, this responsibility 
can become difficult to discharge because (a) many different and evolving legal 
mandates, institutional requirements, professional standards, and ethical con- 
cerns govern the decision of when, how, and to whom test data may be disclosed, 
and (b) these various strictures can differ substantially depending on the setting 
and the purpose of testing and may even conflict with one another. Most recently, 
for instance, the federal Health Insurance Portability and Accountability Act 
(HIPAA) Privacy Rule, which became effective in the United States on April 14, 
2003, has imposed requirements that are raising many questions and creating 
some confusion among testing and assessment professionals as well as other 
health care practitioners. Some answers to frequently asked questions, and fur- 
ther information on how HIPAA affects psychologists, can be found in the APA 
Insurance Trust Web site at http://www.apait.org/resources/hipaa/. 

As a general rule, legal mandates that address the release of test data take 
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precedence over the regulations and concerns of other parties. However, test 
users need to be aware of their duties to (a) protect the confidentiality of test re- 
sults and the security of test materials in specific settings and situations, (b) in- 
form the parties involved about these duties, and (c) try to resolve conflicts be- 
tween various obligations in a manner consistent with the ethical principles and 
standards of their profession. In addition to the Testing Standards (AERA, APA, 
NCME, 1999) and the Evhical Principles of Psychologists and Code of Conduct (APA, 
2002), organized psychology has promulgated a number of documents to help 
psychologists identify and deal with issues related to the security of tests and test 
data and to prevent their misuse. Some of the most pertinent documents of this 
type are listed in Rapid Reference 7.10. 


A New Medium: Testing on the Internet 


The advent and rapid expansion of the Internet in the last two decades is revolu- 
tionizing the field of psychological testing as much as it has affected almost every 
other aspect of contemporary society. Readers may have noticed, scattered 
throughout this book, a fair number of references to Internet sites that provide 
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Information on Safeguarding Tests and Test Data 


In addition to the Testing Standards (AERA, APA, NCME, 1999) and the Ethical 
Principles of Psychologists and Code of Conduct (APA, 2002), both of which deal 
with issues concerning the security of test materials and test data, the American 
Psychological Association (APA) has provided guidance for test users with regard 
to these issues in several other documents. Among the most pertinent of these 
are the following: 


e American Psychological Association, Committee on Legal Issues. (1996). 
Strategies for private practitioners coping with subpoenas or compelled testi- 
mony for client records or test data. Professional Psychology: Research and Prac- 
tes, 27, 249-25, 


e American Psychological Association, Committee on Psychological Tests and As- 
sessment. (1996). Statement on the disclosure of test data. American Psycholo- 
gist 51, 644-648. 


e American Psychological Association, Committee on Psychological Tests and As- 
sessment. (2003). Statement on the use of secure psychological tests in the educa- 
tion of graduate and undergraduate psychology students. Retrieved February 19, 
2003, from http://www.apa.org/science/securetests.htm 


e Test security: Protecting the integrity of tests. (1999). American Psychologist, 54, 
1078. 
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information on psychological tests and testing issues. In addition to these and to 
the many other sources of information about tests and testing currently available 
online, the delivery of psychological testing services on the Internet by a variety 
of providers—some legitimate and many not so—is also well underway. Yet the 
impact that the Internet is likely to have on testing promises to be much greater 
in the coming decades due primarily to the speed with which tests can be devel- 
oped, normed, published, and revised using the capabilities of the World Wide 
Web, as well as to the efficiency and economy with which testing services can be 
delivered online. 

In every aspect of Internet use, whether for commerce, entertainment, infor- 
mation, of communication purposes, the highly democratic and accessible nature 
of the medium poses at once both huge advantages and dangers. As the reader 
may have gathered, with regard to testing, the possibilities the Internet presents 
for harmful uses abound as does the potential for unparalleled advances. Thus, 
the professional examination of the difficulties and benefits inherent in current 
and foreseeable psychological testing practices on the Internet has already begun 
(see, e.g., Buchanan, 2002). One of the most thorough efforts in this regard, up 
to this point, is a report prepared by the APA Task Force on Psychological Test- 


Putting It Into Practice 


How to Test Your Ability to Evaluate Psychological Tests 


A number of Web sites offer surveys, questionnaires, and other tools that pur- 
port to be psychological tests. Almost any search engine will lead you to these 
sites upon entering the terms “psychological tests” or “personality tests.’ Among 
the many offerings in these sites are a variety of instruments, such as “IQ tests,” 
“EQ tests,’ and “career tests,’ that may look like psychological tests and appeal to 
the layperson but have very little or no legitimate scientific bases. Some of these 
so-called tests are offered free of charge; others involve payment of a (typically) 
modest fee of $5, $10, or $15. In many cases, the free instruments do not include 
the full results (or even any results), unless a fee is first paid. Often test takers are 
not informed about the required fee until after they have completed the test. 


Having read this chapter carefully, you should by now be aware that the publish- 
ers of legitimate psychological tests are extremely concerned with the issue of 
test security and do not wish to see their products disseminated to the public at 
arge or used inappropriately. With rare exceptions (e.g., instruments that are in 
the process of development), one is not likely to find serious test publishers or 
authors making their products freely available on the Internet. 


One way to apply some of the knowledge you have gained by reading this book is 
to inspect some of the instruments offered in the Internet and evaluate them with 
regard to (a) whether they meet the criteria that would qualify them as legitimate 
psychological tests and (b) whether the information they provide is of any value. 
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ing on the Internet (Naglieri et al., 2004). This report focuses on a variety of as- 
pects related to the influence of the Internet on psychological testing and assess- 
ment, including psychometric, ethical, and legal issues as well as practical consid- 
erations regarding test security and access, among other things. It also contains 
examples that illustrate current applications of Internet testing and offers 
glimpses into future possibilities afforded by the medium as well as recommen- 
dations for the future. 


CONCLUSION 
Consider the following typical questions and problems: 


* Does my child have an attention-deficit disorder? 

e How should we go about selecting the best applicants for our police de- 
partment? 

e Is this 65-year-old patient suffering from an incipient dementing disor- 
der, or is he depressed due to the death of his wife? 

e Which of these three equally experienced candidates for the position of 
chief financial officer in my company should I hire? 

e What should I major in? 

° Should this convicted sexual predator be released from prison? 

e What kind of therapeutic intervention would be most effective for this 
client? 

e Is this individual feigning disability in order to collect compensation, or 
is she really disabled? 


Although all of these questions and assessment problems can be addressed 
with the aid of tests, none can be answered by a single test score or even a com- 
bination of scores. The determinations that need to be made in these situations, 
and similar ones, require data from multiple sources as well as information about 
the specific objectives of the people involved, the contexts in which the questions 
are raised, and the potential consequences of the decisions entailed in each case. 

It should be clear at this point that the application of psychological tests is 
fraught with the possibility of error at every step. However, so are most other 
human endeavors. Once the essential question to consider (i.e., whether the use 
of tests can contribute to more rational, equitable, beneficial, and responsible 
decision-making in a given situation) is answered affirmatively, what is needed is 
to implement the plan for test use as conscientiously as possible through all of its 
phases. If this is done, the role of tests, especially compared to other tools, is likely 
to prove quite useful. 
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‘a TEST YOURSELF wA. 


|. The first question to consider when psychological testing is contem- 
plated is 


a) whether testing is necessary. 
b) what kind of test to use. 
c) the costs associated with testing. 


2. Which of the following is not a primary advantage attendant to the use of 
psychological tests? 


Objectivity 
Efficiency 


[e3] 


D 


c) Easy availability 


3. All other things being equal, the potential gain in the accuracy of a selec- 
tion decision is greatest when the base rate is closest to 


a) 1.00. 
b) 0.75. 
c) 0.50. 
d) 0.00. 


4. An ideal situation, for purposes of accuracy in selecting employees, would 
involve a base rate,a selection ratio, and a test with a 
degree of validity. 


a) moderate/high/high 
b) moderate/low/high 


c) high/low/moderate 

d) high/high/high 

5. Which one of the following assessment tools is most likely to provide 
valid and reliable evaluative information for individual assessment? 


a) Interviewing 

b) References 

c) Informal observation 
d) Biodata 


6. As a general rule, the presence of third parties, other than the examiner 
and test taker, during the administration of individual tests is 


a) desirable. 
b) undesirable. 


c) neither a nor b. 


(continued ) 
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7. Which of the following is not one of the areas in which computer-based 
test administration offers advantages over individual test administration? 


a) Uniformity of procedure 
b) Cost effectiveness 

c) Precision capabilities 

d) Qualitative data gathering 


8. The perspective on testing presented by the author stresses the fact that 
psychological tests 


a) can be a valuable component in most cases when assessment is needed. 
b) can often be the sole source of information on which to base decisions. 
invariably constitute the most efficient tools available for assessment. 


9. In communicating test results to the consumers of test data, the most 
pertinent information to be conveyed to them is the 


O 


a) numerical score obtained by examinees. 


b) labels or diagnoses derived from test scores. 


a 


meaning of test scores. 


10. The legal and ethical responsibilities of test users with regard to appro- 
priate interpretation of test results is 


a) obviated when they use tests on their own behalf. 

c) obviated when they use tests on behalf of a third party. 

b) obviated when they use tests on behalf of their own clients. 
d) never obviated. 


Answers: |. a; 2.¢; 3.c4.b; 5. d; 6. b; 7. d; 8.a;9.c 10.d. 
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Commercially Available Tests Mentioned in the Text 


Test Name (Abbreviation of Current Version) 
ACT Assessment 

Beck Depression Inventory (BDI) 

Beta II 

Boston Diagnostic Aphasia Examination (BDAS) 
Bracken Basic Concept Scale—Revised (BBCS-R) 
California Psychological Inventory (CPI) 

Clerical Abilities Battery (CAB) 

College-Level Examination Program (CLEP) 
Crawford Small Parts Dexterity Test (CSPDT) 
Das-Naglieri Cognitive Assessment System (CAS) 
Differential Ability Scales (DAS) 

Graduate Record Exam (GRE) 

Halstead-Reitan Neuropsychological Battery (HRNB) 
Infant-Toddler Developmental Assessment (IDA) 
Iowa Test of Basic Skills (TTBS) 

Jackson Vocational Interest Survey (JVIS) 

Kaufman Assessment Battery for Children (K-ABC-II) 
Kaufman Adolescent and Adult Intelligence Test (KAIT) 
Law School Admission Test (LSAT) 

Medical College Admission Test (MCAT) 

Millon Clinical Multiaxial Inventory (MCMI-II) 
Mini-Mental State Examination (MMSE) 


Minnesota Multiphasic Personality Inventory 
(MMPI-2 & MMPI-A) 


Myers-Briggs Type Indicator (MBTT) 
Otis-Lennon School Ability Test (OLSAT 8) 
Quality of Life Inventory (QOLI) 

Revised NEO Personality Inventory (NEO PI-R) 


Publishetr’s Code 


ACT 
TPC 
TPC 
LWW 
TPC 
CPP 
TPC 
TCB 
TPC 
RIV 
TPC 
ETS 
RNL 
RIV 
RIV 
SIG 
AGS 
AGS 
LSAC/LSAS 
AAMC 
PA 
PAR 


UMP/PA 
CPP 
HEM 

PA 

PAR 


Appendix B contains the complete names and internet addresses of the test publishers 


listed in here, arranged in alphabetical order according to the Publishers’ codes used in this Ap- 


pendix. 
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Test Name (Abbreviation of Current Version) 
Rorschach 

SAT (formerly known as Scholastic Aptitude Test) 
Stanford-Binet Intelligence Scale (S-B 5) 

Stanford Diagnostic Mathematics Test (SDMT) 
Stanford Diagnostic Reading Test (SDRT) 
State-Trait Anxiety Inventory (STAT) 

Strong Interest Inventory (SII) 

Symptom Checklist-90—Revised (SCL-90-R) 

Test of English as a Foreign Language (TOEFL) 
Test of Variables of Attention (TOVA) 

Thematic Apperception Test (TAT) 

Validity Indicator Profile (VIP) 

Wechsler Adult Intelligence Scale (WAIS-III) 
Wechsler Intelligence Scale for Children (WISC-IV) 
Whitaker Index of Schizophrenic Thinking (WIST) 
Wide Range Achievement Test (WRAT) 

Wonderlic Personnel Test 

Woodcock-Johnson batteries 


WorkKeys Assessments 


Publishetr’s Code 
H&H 
TCB 
RIV 
HEM 
HEM 
Mind 
CPP 
PA 
ETS 
UAD 
HAR 
PA 
TPC 
TPC 
WPS 
WRI 
WON 
RIV 
ACT 
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Addresses of Test Publishers and Distributors 


Publishet’s Name 


Association of American Medical 
Colleges 


ACT, Inc. 

American Guidance Service 
Consulting Psychologists Press 
Educational Testing Service 
Harvard University Press 


Harcourt Educational 
Measutement 


Hogrefe & Huber Publishers 


Law School Admission Council/Law 
School Admission Service 


Lippincott Williams & Wilkins 
Mind Garden, Inc. 
Pearson Assessments (formerly NCS) 


Psychological Assessment 
Resources 


Reitan Neuropsychological Laboratory 
Riverside Publishing 

Sigma Assessment Systems, Inc. 

The College Board 

The Psychological Corporation 
Universal Attention Disorders, Inc. 
University of Minnesota Press 
Wondetlic, Inc. 

Western Psychological Services 

Wide Range, Inc. 


Address 


www.aamc.org 


www.act.org 
www.agsnet.com 
www.cpp.com 
www.ets.org 
www.hup.harvard.edu 
www. HEMWEB.com 


www.hhpub.com 


wwwa.lsac.org 


www.lww.com 
www.mindgarden.com 
www.pearsonassessments.com 


www.patinc.com 


www.teitanlabs.com 
www.tivetsidepublishing.com 
www.sigmaassessmentsystems.com 
www.collegeboard.com 
www.PsychCorp.Com 
www.tovatest.com 
www.upress.umn.edu/tests 
www.wonderlic.com 
www.wpspublish.com 


www.widerange.com 


See Appendix A for publishers’ codes. 
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Table of Areas and Ordinates of the Normal Curve 


EXPLANATION OF THE TABLE 


Column (1) lists standard scores (i.e., z scores) from 0.00 to 3.24 (at intervals of 
.01) and from 3.30 to 3.70 (at intervals of .10). 


= a C.1 
Z score . 
Oo ( ) 


where 


x = the distance between any point on the baseline and the ean of the 
distribution 
O (sigma) = the standard deviation of the distribution 


The mean of z scores is zero; the standard deviation of z scores is 1. 

Column (2) lists the proportion of the area of the curve comprised in the seg- 
ment between the mean and any of the z scores. Since the normal curve is per- 
fectly symmetrical, when z = 0 (at the mean), one half of the curve (.5000, or 50%) 
is above z and one half is below z. 

When the curve is divided at any point other than the mean, there will be a 
larger area, listed in Column (3) and a smaller area, listed in Column (4). If the point 
that divides the curve is to the /e/t of the mean, the z score has a negative sign and 
the smaller area is to its left; if the point that divides the curve is to the right of the 
mean, the z score is positive and the smaller area is to its right. 

Column (5) lists the y ordinate values, or the height of the curve, at every z 
score point. 


HOW TO USE THE TABLE 
Relationship Between z Scores and the Areas of the Curve 


Figure C.1, Panel A, displays the distance between a z score of +1.50 and the 
mean. If the z score of 1.50 is found in Column (1) of the table, Column (2) shows 
that the area between it and the mean is .4332, or 43.32% of the curve. Since the 
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A. 
5000 4332 
X 
z scores 0 +1.50 
Mean 
B. 
z scores -1.50 0 
Mean 
C. 
.9332 
.0668 
Z scores 0 +1.50 
Mean 
D. 
.9332 
y 
-0668 | 
z scores -1.50 0 
Mean 


Figure C.I Areas of the normal curve 
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curve is symmetrical, any given z score subtends the same area from the mean, 
whether it is positive (above the mean) or negative (below the mean). Therefore, 
in Figure C.1, Panel B, a z score of —1.50 again subtends an area of .4332 between 
the mean and z. To find the proportion, or percentage, of the area that falls above 
az of +1.50, we subtract .4332 from .5000 and get .0668, or 6.68%. To find the 
area belowa z of +1.50, we add .4332 to .5000 and get .9332, or 93.32%. These val- 
ues ate shown in Panel C of Figure C.1. Panel D of the figure shows the results 
with a z score of —1.50. Here, as is the case with all negative z scores in a normal 
curve, the larger portion is above the z and is thus found by adding the value in 
column (2) of the table to .5000 (.4332 + .5000 = .9332); the smaller portion falls 
below z and is found by subtracting .4332 from .5000, which results in .0668. Col- 
umns (3) and (4) of the table provide the results of these computations. To verify 
the results, find the entries for a z score of 1.50 in columns (3) and (4). For fur- 
ther practice, corroborate the ateas of the curve shown in Figure 2.2 of Chapter 
2 for © values or z scores of +1, +2, and +3. 


Using the Table for Hypothesis Testing 


When the normal curve is applied to hypothesis testing, in inferential statistics, it 
is used to ascertain the likelihood that the critical value (z) obtained could have 
resulted by chance. Since z values give the proportion of the area in only one end 
of the curve, when hypotheses allow the possibility of variation in two directions, 
the proportion of the area that falls beyond the critical z value has to be doubled 
in order to find the probability level associated with the critical value obtained. 


Example: Suppose we are trying to find out whether a 10-point difference 
that has been obtained between the two IQ scores of an individual (e.g., a 
Verbal IQ of 115 and a Performance IQ of 105) is statistically significant, in 
a situation where neither IQ is expected to be higher than the other. To test 
the null hypothesis of no difference, we obtain the critical z value for the 
obtained difference of 10 points by dividing that difference by the standard 
error of the difference (SE ø) between the scores, which is a statistic de- 
rived from the respective reliabilities of the Verbal and Performance scales 
(see Chapter 4). In this example, let us assume that the SE g = 5; thus, the 
critical z ratio is 10 + 5 = 2.00. The area beyond that critical z value, 0.0228, 
represents the probability (p) level for the obtained difference. However, 
because the score difference could have occurred in either direction, p is 
doubled (0.0228 x 2 = 0.0456) to obtain the likelihood (4.56%) that a 10- 
point difference between the scores could have been obtained if there were 
no difference between the two IQ scores. 
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This example describes a /wo-tailed test, which is the typical way in which the sig- 
nificance of various findings is tested. If there is a specific directional hypothesis, 
as there might be in an experiment where one expects results to be ina certain di- 
rection, a one-tailed test is performed. In such cases, if the results do fall in the ex- 
pected direction, the p level for the critical value does not have to be doubled. 


Table C.I Table of Areas and Ordinates of the Normal Curve in Terms of 
Standard Scores z = x/o,,, 


Standard Score Area from Area in Area in J 
Z = x/0x Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
0.00 0000 5000 5000 3989 
0.01 .0040 5040 4960 3989 
0.02 .0080 5080 4920 3989 
0.03 0120 5120 4880 3988 
0.04 .0160 5160 4840 3986 
0.05 0199 5199 4801 3984 
0.06 0239 5239 4761 3982 
0.07 0279 5279 A721 3980 
0.08 0319 5319 4681 3977 
0.09 0359 5359 4641 3973 
0.10 0398 5398 4602 3970 
0.11 0438 5438 4562 3965 
0.12 0478 5478 4522 3961 
0.13 0517 5517 4483 3956 
0.14 0557 5557 4443 3951 
0.15 0596 5596 4404 3945 
0.16 .0636 5636 4364 3939 
0.17 0675 5675 4325 3932 
0.18 0714 5714 4286 3925 
0.19 0753 5753 4247 3918 
0.20 0793 5793 4207 3910 
0.21 0832 5832 4168 3902 
0.22 0871 5871 4129 3894 
0.23 .0910 5910 4090 3885 
0.24 0948 5948 4052 3876 
0.25 0987 5987 4013 3867 


(continued) 
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Table C.I Continued 


Standard Score Area from Area in Area in J 
Z = x/0x Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
0.26 1026 6026 3974 3857 
0.27 1064 6064 3936 3847 
0.28 1103 .6103 3897 3836 
0.29 1141 6141 3859 3825 
0.30 1179 6179 3821 3814 
0.31 1217 6217 3783 3802 
0.32 1255 6255 3745 3790 
0.33 1293 6293 3707 3778 
0.34 1331 6331 3669 3765 
0.35 1368 .6368 3632 3752 
0.36 1406 6406 3594 3739 
0.37 1443 6443 3557 3725 
0.38 1480 6480 3520 3712 
0.39 1517 6517 3483 3697 
0.40 1554 6554 3446 3683 
0.41 1591 659 3409 3668 
0.42 1628 .6628 3372 3653 
0.43 1664 .6664 3336 3637 
0.44 .1700 .6700 3300 3621 
0.45 1736 .6736 3264 3605 
0.46 1772 .6772 3228 3589 
0.47 .1808 .6808 3192 3572 
0.48 1844 6844 3156 3555 
0.49 1879 .6879 3121 3538 
0.50 1915 6915 3085 3521 
0.51 1950 6950 3050 3503 
0.52 1985 6985 3015 3485 
0.53 2019 .7019 2981 3467 
0.54 2054 £7054 2946 3448 
0.55 2088 .7088 2912 3429 
0.56 2123 .7123 .2877 3410 
0.57 2157 .7157 2843 3391 
0.58 2190 .7190 2810 3372 


0.59 2224 £7224 .2776 3352 
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Table C.I Continued 


Standard Score Area from Area in Area in J 
zZ=x/Oy Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
0.60 2257 £7257 2743 3332 
0.61 2291 7291 .2709 3312 
0.62 2324 £7324 .2676 3292 
0.63 2357 .7357 2643 3271 
0.64 2389 .7389 2611 3251 
0.65 2422 £7422 2578 3230 
0.66 2454 £7454 2546 3209 
0.67 2486 .7486 2514 3187 
0.68 2517 £7517 2483 3166 
0.69 2549 £7549 2451 3144 
0.70 2580 .7580 2420 3123 
0.71 2611 7611 2389 3101 
0.72 2642 .7642 2358 3079 
0.73 .2673 .7673 2327 3056 
0.74 .2704 .7704 2296 3034 
0.75 2734 .7734 2266 3011 
0.76 .2764 .7764 2236 2989 
0.77 2794 .7794 2206 .2966 
0.78 2823 .7823 2177 2943 
0.79 2852 £7852 2148 2920 
0.80 2881 .7881 2119 2897 
0.81 .2910 .7910 .2090 2874 
0.82 2939 .7939 2061 2850 
0.83 .2967 .7967 2033 2827 
0.84 2995 7995 2005 .2803 
0.85 3023 8023 1977 .2780 
0.86 3051 8051 1949 .2756 
0.87 3078 .8078 1922 2732 
0.88 3106 8106 1894 .2709 
0.89 3133 8133 .1867 2685 
0.90 3159 8159 184 .2661 
0.91 3186 8186 1814 .2637 
0.92 3212 8212 1788 .2613 


(continued) 
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Table C.I Continued 


Standard Score Area from Area in Area in J 

zZ=x/Oy Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
0.93 3238 8238 1762 2589 
0.94 3264 8264 1736 2565 
0.95 3289 8289 1711 2541 
0.96 3315 8315 1685 .2516 
0.97 3340 8340 1660 2492 
0.98 3365 8365 1635 .2468 
0.99 3389 8389 1611 2444 
.00 3413 8413 1587 2420 
01 3438 8438 1562 .2396 
02 3461 8461 1539 2371 
03 3485 8485 1515 2347 
04 3508 8508 1492 2323 
£05 3531 8531 1469 2299 
.06 3554 8554 1446 2275 
.07 .3577 .8577 .1423 2201 
08 3599 8599 1401 2227 
.09 3621 8621 1379 .2203 
.10 3643 8643 1357 2179 
.11 .3665 .8665 .1335 :2155 
.12 .3686 .8686 .1314 .2131 
13 3708 .8708 1292 .2107 
14 3729 8729 1271 .2083 
615 3749 8749 1251 2059 
16 3770 .8770 1230 .2036 
.17 .3790 .8790 .1210 .2012 
.18 .3810 .8810 .1190 .1989 
9 .3830 .8830 .1170 .1965 
.20 .3849 .8849 .1151 .1942 
.21 .3869 .8869 .1131 .1919 
22 3888 8888 1112 1895 
23 3907 8907 1093 1872 
24 3925 8925 1075 1849 
325 .3944 .8944 .1056 .1826 
.26 .3962 .8962 .1038 .1804 


Table C.I Continued 
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Standard Score Area from Area in Area in y 

Z=x/Ox Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
2} 3980 8980 1020 1781 
28 3997 8997 1003 1758 
29 4015 9015 0985 1736 
30 4032 9032 .0968 1714 
31 4049 9049 0951 1691 
32 4066 .9066 0934 1669 
33 4082 9082 0918 1647 
34 4099 9099 0901 1626 
35 4115 9115 0885 1604 
36 4131 9131 0869 1582 
37 4147 9147 0853 1561 
38 4162 9162 0838 1539 
39 177 OTT .0823 .1518 
.40 .4192 .9192 .0808 .1497 
Al 4207 .9207 .0793 1476 
42 4222 9222 .0778 1456 
A3 4236 9236 .0764 1435 
44 4251 9251 .0749 1415 
A5 4265 9265 0735 1394 
46 4279 9279 072 1374 
AT 4292 9292 .0708 1354 
48 4306 9306 0694 1334 
49 4319 9319 .068 1315 
50 4332 9332 .0668 1295 
51 4345 9345 0655 1276 
52 4357 9357 0643 1257 
53 4370 .9370 0630 1238 
54 4382 9382 .0618 1219 
55 4394 9394 .0606 1200 
56 4406 9406 0594 1182 
57 4418 9418 0582 1163 
58 4429 9429 0571 1145 
59 A441 9441 0559 1127 


(continued) 
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Table C.I Continued 


Standard Score Area from Area in Area in y 
Z = x/0x Mean to z Larger Portion Smaller Portion at x/0x 

(1) (2) (3) (4) (5) 

.60 4452 9452 0548 1109 
61 4463 9463 .0537 1092 
62 4474 9474 0526 1074 
63 4484 9484 .0516 1057 
64 4495 9495 0505 1040 
65 4505 9505 0495 1023 
66 4515 9515 0485 1006 
.67 4525 9525 0475 0989 
68 4535 9535 0465 .0973 
.69 4545 9545 0455 .0957 
.70 4554 9554 0446 0940 
71 4564 9564 0436 0925 
72 4573 9573 0427 .0909 
73 4582 9582 0418 0893 
.74 4591 9591 0409 .0878 
75 4599 9599 0401 0863 
.76 4608 .9608 0392 0848 
77 4616 .9616 0384 0833 
.78 4625 9625 0375 0818 
.79 4633 .9633 .0367 0804 
80 4641 9641 0359 .0790 
81 4649 .9649 0351 .0775 
82 4656 .9656 0344 0761 
83 4664 .9664 0336 .0748 
84 671 9671 0329 .0734 
85 4678 .9678 0322 0721 
86 4686 .9686 0314 .0707 
87 4693 .9693 .0307 0694 
88 4699 .9699 .030 0681 
89 4706 .9706 0294 0669 
90 4713 9713 0287 0656 
91 4719 9719 028 0644 
92 4726 .9726 0274 0632 
93 4732 9732 0268 .0620 


C APPENDIXC 303 


Table C.I Continued 


Standard Score Area from Area in Area in y 
Z = x/0x Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
1.94 4738 .9738 0262 .0608 
1.95 4744 9744 0256 0596 
1.96 4750 9750 0250 0584 
1.97 4756 .9756 0244 .0573 
1.98 A761 .9761 0239 0562 
1.99 4767 .9767 0233 0551 
2.00 A772 9772 0228 0540 
2.01 4778 .9778 0222 .0529 
2.02 4783 9783 .0217 0519 
2.03 4788 .9788 0212 0508 
2.04 4793 9793 .0207 0498 
2.05 4798 .9798 0202 0488 
2.06 4803 .9803 0197 0478 
2.07 .4808 .9808 0192 0468 
2.08 4812 9812 .0188 0459 
2.09 4817 9817 0183 0449 
2.10 4821 9821 .0179 0440 
2.11 4826 .9826 0174 0431 
2.12 4830 .9830 .0170 0422 
2.13 4834 9834 .0166 0413 
2.14 4838 .9838 0162 0404 
2.15 4842 9842 0158 .0396 
2.16 4846 .9846 0154 .0387 
2.17 4850 9850 0150 .0379 
2.18 4854 9854 0146 .0371 
2.19 4857 9857 0143 .0363 
2.20 4861 .986 0139 0355 
2.21 .4864 .9864 .0136 .0347 
2.22 .4868 .9868 .0132 .0339 
2.23 4871 .987 0129 0332 
2.24 4875 .9875 0125 0325 
2.25 4878 .9878 0122 0317 
2.26 4881 .9881 .0119 .0310 


(continued) 
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Table C.I Continued 


Standard Score Area from Area in Area in y 
Z=x/Ox Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
2.27 4884 9884 0116 .0303 
2.28 4887 .9887 0113 0297 
2.29 4890 .9890 0110 0290 
2.30 4893 9893 0107 0283 
2.31 4896 .9896 0104 .0277 
2.32 4898 9898 .0102 .0270 
2.33 4901 9901 0099 0264 
2.34 4904 9904 .0096 0258 
2.35 4906 .9906 0094 0252 
2.36 4909 9909 009 0246 
2.37 4911 9911 .0089 0241 
2.38 4913 9913 0087 0235 
2.39 4916 9916 0084 0229 
2.40 4918 9918 0082 0224 
2.41 4920 9920 .0080 0219 
2.42 4922 9922 0078 0213 
2.43 4925 9925 0075 0208 
2.44 4927 9927 .0073 .0203 
2.45 4929 9929 .0071 0198 
2.46 4931 9931 0069 0194 
2.47 4932 9932 .0068 0189 
2.48 4934 9934 0066 0184 
2.49 4936 9936 0064 0180 
2.50 4938 .9938 0062 .0175 
2.51 .4940 .9940 .0060 .0171 
2.52 A941 9941 0059 .0167 
2.53 4943 9943 0057 0163 
2.54 4945 9945 0055 0158 
2.55 4946 9946 0054 0154 
2.56 4948 9948 0052 0151 
2.57 4949 9949 0051 .0147 
2.58 4951 9951 0049 0143 
2.59 4952 9952 .0048 0139 
2.60 4953 9953 0047 0136 


2.61 4955 9955 0045 0132 
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Table C.I Continued 


Standard Score Area from Area in Area in y 
Z = x/0x Mean to z Larger Portion Smaller Portion at x/0x 
(1) (2) (3) (4) (5) 
2.62 4956 9956 0044 0129 
2.63 4957 9957 .0043 0126 
2.64 4959 9959 0041 0122 
2.65 4960 .9960 .0040 0119 
2.66 4961 9961 0039 .0116 
2.67 4962 9962 0038 0113 
2.68 4963 .9963 .0037 0110 
2.69 4964 9964 0036 .0107 
2.70 4965 9965 0035 0104 
2.71 4966 .9966 0034 0101 
2.72 4967 .9967 .0033 0099 
2.73 4968 .9968 0032 0096 
2.74 4969 .9969 0031 0093 
2.75 4970 .9970 .0030 0091 
2.76 A971 9971 0029 0088 
2.77 4972 9972 0028 .0086 
2.78 4973 9973 0027 0084 
2.79 4974 9974 0026 0081 
2.80 4974 9974 0026 0079 
2.81 4975 9975 .0025 .0077 
2.82 .4976 .9976 .0024 .0075 
2.83 A977 9977 0023 .0073 
2.84 A977 9977 0023 0071 
2.85 4978 .9978 0022 0069 
2.86 4979 9979 0021 .0067 
2.87 4979 9979 0021 0065 
2.88 4980 9980 .0020 .0063 
2.89 4981 9981 0019 0061 
2.90 4981 9981 0019 0060 
2.91 4982 9982 0018 0058 
2.92 4982 9982 0018 0056 
2.93 4983 9983 0017 0055 
2.94 4984 9984 0016 0053 
2.95 4984 9984 0016 0051 


(continued ) 
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Table C.I Continued 


Standard Score Area from Area in Area in y 
Z=x/Ox Mean to z Larger Portion Smaller Portion at x/O>» 
(1) (2) (3) (4) (5) 
2.96 4985 9985 0015 0050 
2.97 4985 9985 0015 0048 
2.98 4986 .9986 0014 0047 
2.99 4986 .9986 0014 0046 
3.00 4987 9987 0013 0044 
3.01 4987 9987 0013 0043 
3.02 4987 9987 0013 0042 
3.03 4988 .9988 0012 0040 
3.04 4988 .9988 0012 0039 
3.05 4989 9989 0011 .0038 
3.06 4989 9989 0011 .0037 
3.07 4989 9989 0011 .0036 
3.08 4990 9990 0010 0035 
3.09 4990 9990 0010 0034 
3.10 4990 9990 0010 .0033 
3.11 4991 9991 0009 0032 
3.12 4991 9991 0009 0031 
3.13 4991 9991 0009 .0030 
3.14 4992 9992 0008 0029 
3.15 4992 9992 0008 0028 
3.16 4992 9992 0008 .0027 
3.17 .4992 .9992 .0008 .0026 
3.18 .4993 .9993 .0007 .0025 
3.19 .4993 .9993 .0007 .0025 
3.20 .4993 .9993 .0007 .0024 
3.21 .4993 .9993 .0007 .0023 
3.22 4994 9994 0006 0022 
3.23 4994 9994 0006 0022 
3.24 4994 9994 0006 0021 
3.30 4995 9995 0005 .0017 
3.40 4997 9997 0003 0012 
3.50 4998 9998 0002 0009 
3.60 4998 9998 0002 .0006 
3.70 4999 9999 0001 .0004 


Note: The entries in this table were generated using a computer program. 
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